[2025-11-12 22:10:02,368][mllm.models.large_language_model_local][INFO] - Initializing adapter 'agent_adapter': no initial weights provided or found; starting from scratch.
[2025-11-12 22:10:03,178][mllm.models.adapter_training_wrapper][INFO] - Adapter 'agent_adapter': initialized with fresh weights (no initial weights found).
[2025-11-12 22:10:03,185][mllm.models.large_language_model_local][INFO] - Initializing adapter 'critic_adapter': no initial weights provided or found; starting from scratch.
[2025-11-12 22:10:04,272][mllm.models.adapter_training_wrapper][INFO] - Adapter 'critic_adapter': initialized with fresh weights (no initial weights found).
[2025-11-12 22:12:16,732][__main__][INFO] - Starting iteration 0.
[2025-11-12 22:12:16,736][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:12:16,736][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:12:24,825][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:12:24,908][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given the per-item values, I see that despite having the same value for hats (10) as Bob, I have a much lower value for books (1) compared to his (10), and the same for balls (1). By proposing to take all 10 hats, I can maximize my immediate score since hats are the item with the highest value for me. This strategy aims to secure the highest possible points in this round without risking the allocation being reduced proportionally due to over-proposals. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:12:31,621][__main__][INFO] - Number of regex retries in iteration 0: 2
[2025-11-12 22:12:31,622][__main__][INFO] - agents played in iteration 0 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:12:44,265][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-12 22:12:44,288][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-12 22:12:44,311][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-12 22:12:44,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-12 22:12:44,333][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:12:44,333][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:12:44,889][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:12:45,684][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:12:46,180][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:12:46,677][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:12:47,170][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:12:47,668][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:12:48,164][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:12:48,659][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:12:49,151][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:12:49,643][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:12:50,139][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:12:50,630][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:12:51,123][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:12:51,614][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:12:52,105][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:12:52,601][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:12:53,113][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:12:53,609][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:12:54,103][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:12:54,602][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:12:55,101][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:12:55,596][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:12:56,127][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:12:56,626][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:12:57,127][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:12:57,621][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:12:58,112][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:12:58,604][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:12:59,099][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:12:59,605][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:13:00,096][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:13:00,587][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:13:01,092][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:13:01,584][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:13:02,081][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:13:02,584][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:13:03,080][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:13:03,578][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:13:04,072][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:13:04,566][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:13:05,062][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:13:05,559][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:13:06,052][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:13:06,545][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:13:07,041][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:13:07,537][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:13:08,026][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:13:08,519][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:13:09,010][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:13:09,502][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:13:10,005][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:13:10,500][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:13:10,994][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:13:11,504][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:13:11,996][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:13:12,491][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:13:12,991][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:13:13,481][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:13:13,976][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:13:14,470][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:13:14,963][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:13:15,456][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:13:15,951][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:13:16,446][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:13:16,940][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9701 tokens.
[2025-11-12 22:13:17,675][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 15.77%, Current % of VRAM taken: 53.22%, Block Peak % of device VRAM: 61.00%, ΔTime: 00:00:32
[2025-11-12 22:13:18,548][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:13:18,550][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:13:18,552][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:13:19,697][__main__][INFO] - Iteration 1 took 1m 2s (23.64% Gen, 74.54% Train). Generation: 14s, Training: 46s. Estimated remaining time: 52h 24m 28s. Estimated total time: 52h 28m 6s. Time estimates for 10 more iterations: 10m 29s, 100 more iterations: 1h 44m 56s, 500 more iterations: 8h 44m 41s.
[2025-11-12 22:13:19,700][__main__][INFO] - Starting iteration 1.
[2025-11-12 22:13:20,125][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:13:20,127][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:13:32,766][__main__][INFO] - Number of regex retries in iteration 1: 0
[2025-11-12 22:13:32,767][__main__][INFO] - agents played in iteration 1 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:13:33,760][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 46.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:13:33,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 46.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:13:33,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 46.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:13:33,834][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 46.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:13:33,835][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:13:33,835][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:13:34,476][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:13:34,935][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:13:35,429][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:13:35,940][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:13:36,434][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:13:36,924][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:13:37,414][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:13:37,904][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:13:38,397][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:13:38,889][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:13:39,379][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:13:39,867][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:13:40,361][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:13:40,848][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:13:41,342][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:13:41,829][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:13:42,321][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:13:42,811][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:13:43,301][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:13:43,791][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:13:44,282][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:13:44,772][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:13:45,261][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:13:45,752][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:13:46,241][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:13:46,730][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:13:47,219][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:13:47,708][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:13:48,195][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:13:48,684][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:13:49,173][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:13:49,662][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:13:50,151][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:13:50,639][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:13:51,127][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:13:51,617][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:13:52,124][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:13:52,612][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:13:53,101][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:13:53,603][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:13:54,095][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:13:54,585][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:13:55,074][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:13:55,564][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:13:56,059][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:13:56,550][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:13:57,042][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:13:57,529][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:13:58,020][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:13:58,511][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:13:59,002][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:13:59,493][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:13:59,985][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:14:00,475][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:14:00,965][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:14:01,457][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:14:01,946][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:14:02,440][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:14:02,929][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:14:03,422][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:14:03,914][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:14:04,406][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:14:04,899][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:14:05,391][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:14:05,884][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9634 tokens.
[2025-11-12 22:14:06,582][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.97%, Current % of VRAM taken: 60.23%, Block Peak % of device VRAM: 61.59%, ΔTime: 00:00:32
[2025-11-12 22:14:07,301][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:14:07,303][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:14:07,305][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:14:08,396][__main__][INFO] - Iteration 2 took 48s (26.18% Gen, 71.55% Train). Generation: 12s, Training: 34s. Estimated remaining time: 40h 9m 9s. Estimated total time: 40h 13m 35s. Time estimates for 10 more iterations: 8m 2s, 100 more iterations: 1h 20m 27s, 500 more iterations: 6h 42m 15s.
[2025-11-12 22:14:08,398][__main__][INFO] - Starting iteration 2.
[2025-11-12 22:14:08,878][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:14:08,879][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:14:12,951][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Explanation: Given the values, you have a significant advantage in hats compared to Bob. Since Bob values books and balls more highly, we can secure all hats for ourselves to maximize our points. This approach aligns with your high value for hats and your goal of maximizing points. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:14:22,919][__main__][INFO] - Number of regex retries in iteration 2: 1
[2025-11-12 22:14:22,919][__main__][INFO] - agents played in iteration 2 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:14:23,852][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:14:23,880][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:14:23,907][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:14:23,930][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:14:23,930][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:14:23,931][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:14:24,529][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:14:24,982][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:14:25,478][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:14:25,976][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:14:26,469][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:14:26,966][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:14:27,461][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:14:27,955][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:14:28,452][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:14:28,955][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:14:29,460][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:14:29,968][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:14:30,477][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:14:30,982][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:14:31,475][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:14:31,969][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:14:32,463][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:14:32,956][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:14:33,446][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:14:33,943][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:14:34,435][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:14:34,925][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:14:35,420][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:14:35,911][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:14:36,402][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:14:36,891][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:14:37,380][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:14:37,870][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:14:38,360][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:14:38,851][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:14:39,341][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:14:39,830][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:14:40,333][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:14:40,823][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:14:41,313][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:14:41,805][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:14:42,297][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:14:42,787][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:14:43,280][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:14:43,774][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:14:44,268][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:14:44,759][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:14:45,249][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:14:45,750][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:14:46,241][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:14:46,730][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:14:47,237][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:14:47,726][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:14:48,215][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:14:48,710][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:14:49,200][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:14:49,689][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:14:50,179][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:14:50,668][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:14:51,160][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:14:51,649][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:14:52,138][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:14:52,640][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:14:53,134][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:14:53,625][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:14:54,121][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:14:54,614][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:14:55,132][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:14:55,627][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:14:56,125][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9670 tokens.
[2025-11-12 22:14:56,823][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.96%, Current % of VRAM taken: 60.22%, Block Peak % of device VRAM: 61.64%, ΔTime: 00:00:32
[2025-11-12 22:14:57,572][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:14:57,574][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:14:57,576][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:14:58,504][__main__][INFO] - Iteration 3 took 49s (28.29% Gen, 69.83% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 16m 5s. Estimated total time: 41h 21m 22s. Time estimates for 10 more iterations: 8m 16s, 100 more iterations: 1h 22m 42s, 500 more iterations: 6h 53m 33s.
[2025-11-12 22:14:58,506][__main__][INFO] - Starting iteration 3.
[2025-11-12 22:14:58,961][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:14:58,962][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:15:12,042][__main__][INFO] - Number of regex retries in iteration 3: 0
[2025-11-12 22:15:12,043][__main__][INFO] - agents played in iteration 3 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:15:12,949][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:15:12,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:15:13,004][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:15:13,027][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:15:13,027][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:15:13,028][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:15:13,638][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:15:14,088][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:15:14,586][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:15:15,082][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:15:15,574][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:15:16,072][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:15:16,562][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:15:17,078][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:15:17,571][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:15:18,062][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:15:18,553][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:15:19,042][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:15:19,532][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:15:20,038][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:15:20,528][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:15:21,018][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:15:21,520][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:15:22,010][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:15:22,502][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:15:22,994][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:15:23,486][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:15:23,979][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:15:24,474][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:15:24,965][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:15:25,455][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:15:25,945][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:15:26,447][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:15:26,937][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:15:27,427][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:15:27,920][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:15:28,411][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:15:28,907][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:15:29,403][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:15:29,895][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:15:30,390][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:15:30,881][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:15:31,374][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:15:31,867][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:15:32,358][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:15:32,848][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:15:33,341][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:15:33,834][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:15:34,340][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:15:34,831][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:15:35,324][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:15:35,825][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:15:36,316][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:15:36,807][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:15:37,298][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:15:37,787][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:15:38,279][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:15:38,768][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:15:39,258][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:15:39,750][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:15:40,241][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:15:40,743][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:15:41,231][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:15:41,721][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:15:42,211][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:15:42,705][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:15:43,199][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:15:43,692][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:15:44,184][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:15:44,677][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:15:45,171][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9722 tokens.
[2025-11-12 22:15:45,866][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.92%, Current % of VRAM taken: 60.18%, Block Peak % of device VRAM: 61.59%, ΔTime: 00:00:32
[2025-11-12 22:15:46,581][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:15:46,583][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:15:46,584][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:15:47,546][__main__][INFO] - Iteration 4 took 48s (26.92% Gen, 71.09% Train). Generation: 13s, Training: 34s. Estimated remaining time: 40h 23m 10s. Estimated total time: 40h 29m 16s. Time estimates for 10 more iterations: 8m 5s, 100 more iterations: 1h 20m 58s, 500 more iterations: 6h 44m 52s.
[2025-11-12 22:15:47,548][__main__][INFO] - Starting iteration 4.
[2025-11-12 22:15:47,995][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:15:47,996][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:15:48,747][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:16:00,374][__main__][INFO] - Number of regex retries in iteration 4: 1
[2025-11-12 22:16:00,375][__main__][INFO] - agents played in iteration 4 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:16:01,271][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.23%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:16:01,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.23%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:16:01,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.23%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:16:01,342][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.23%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:16:01,343][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:16:01,344][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:16:01,984][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:16:02,437][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:16:02,946][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:16:03,440][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:16:03,930][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:16:04,423][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:16:04,912][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:16:05,402][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:16:05,894][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:16:06,385][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:16:06,878][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:16:07,368][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:16:07,859][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:16:08,365][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:16:08,856][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:16:09,346][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:16:09,839][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:16:10,329][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:16:10,818][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:16:11,307][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:16:11,799][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:16:12,292][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:16:12,781][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:16:13,273][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:16:13,763][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:16:14,256][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:16:14,754][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:16:15,242][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:16:15,732][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:16:16,237][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:16:16,725][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:16:17,215][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:16:17,714][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:16:18,205][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:16:18,695][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:16:19,185][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:16:19,674][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:16:20,165][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:16:20,653][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:16:21,142][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:16:21,636][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:16:22,124][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:16:22,616][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:16:23,107][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:16:23,596][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:16:24,087][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:16:24,576][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:16:25,066][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:16:25,554][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:16:26,041][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:16:26,556][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:16:27,045][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:16:27,533][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:16:28,031][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:16:28,520][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:16:29,010][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:16:29,500][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:16:29,989][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:16:30,479][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:16:30,966][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:16:31,460][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:16:31,950][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:16:32,440][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:16:32,935][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:16:33,426][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9568 tokens.
[2025-11-12 22:16:34,132][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.88%, Current % of VRAM taken: 60.14%, Block Peak % of device VRAM: 61.59%, ΔTime: 00:00:32
[2025-11-12 22:16:34,861][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:16:34,863][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:16:34,865][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:16:35,807][__main__][INFO] - Iteration 5 took 47s (25.89% Gen, 72.14% Train). Generation: 12s, Training: 34s. Estimated remaining time: 39h 43m 45s. Estimated total time: 39h 50m 39s. Time estimates for 10 more iterations: 7m 58s, 100 more iterations: 1h 19m 41s, 500 more iterations: 6h 38m 26s.
[2025-11-12 22:16:35,809][__main__][INFO] - Starting iteration 5.
[2025-11-12 22:16:36,241][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:16:36,241][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:16:40,774][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:16:41,330][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given Alice's per-item values, she values hats much less than I do, while she values books much more than I do. To maximize my points, I should keep as many hats as possible since my value for hats is significantly higher than any other item. I will not propose any books or balls as Alice will likely take all of these given her higher values for books and balls. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:16:41,777][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:16:50,485][__main__][INFO] - Number of regex retries in iteration 5: 3
[2025-11-12 22:16:50,486][__main__][INFO] - agents played in iteration 5 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:16:51,387][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.24%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:16:51,414][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.24%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:16:51,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.24%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:16:51,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.24%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:16:51,464][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:16:51,465][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:16:52,092][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:16:52,542][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:16:53,033][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:16:53,525][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:16:54,016][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:16:54,506][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:16:54,994][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:16:55,485][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:16:55,982][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:16:56,475][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:16:56,976][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:16:57,468][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:16:57,960][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:16:58,454][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:16:58,945][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:16:59,437][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:16:59,943][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:17:00,433][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:17:00,924][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:17:01,415][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:17:01,908][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:17:02,401][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:17:02,892][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:17:03,382][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:17:03,872][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:17:04,361][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:17:04,865][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:17:05,354][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:17:05,844][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:17:06,360][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:17:06,847][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:17:07,337][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:17:07,826][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:17:08,316][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:17:08,806][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:17:09,295][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:17:09,785][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:17:10,280][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:17:10,771][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:17:11,262][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:17:11,758][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:17:12,249][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:17:12,776][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:17:13,268][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:17:13,765][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:17:14,262][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:17:14,758][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:17:15,250][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:17:15,742][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:17:16,234][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:17:16,725][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:17:17,215][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:17:17,706][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:17:18,200][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:17:18,689][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:17:19,182][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:17:19,673][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:17:20,164][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:17:20,678][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:17:21,172][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:17:21,667][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:17:22,163][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:17:22,659][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:17:23,168][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:17:23,666][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9710 tokens.
[2025-11-12 22:17:24,360][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 61.67%, ΔTime: 00:00:32
[2025-11-12 22:17:25,126][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:17:25,128][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:17:25,130][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:17:26,065][__main__][INFO] - Iteration 6 took 49s (28.59% Gen, 69.53% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 23m 30s. Estimated total time: 41h 31m 14s. Time estimates for 10 more iterations: 8m 18s, 100 more iterations: 1h 23m 2s, 500 more iterations: 6h 55m 12s.
[2025-11-12 22:17:26,067][__main__][INFO] - Starting iteration 6.
[2025-11-12 22:17:26,584][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:17:26,584][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:17:39,563][__main__][INFO] - Number of regex retries in iteration 6: 0
[2025-11-12 22:17:39,563][__main__][INFO] - agents played in iteration 6 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:17:40,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:17:40,503][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:17:40,528][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:17:40,551][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:17:40,552][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:17:40,552][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:17:41,181][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:17:41,638][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:17:42,136][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:17:42,627][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:17:43,124][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:17:43,617][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:17:44,109][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:17:44,602][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:17:45,092][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:17:45,592][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:17:46,085][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:17:46,577][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:17:47,070][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:17:47,563][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:17:48,059][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:17:48,551][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:17:49,046][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:17:49,544][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:17:50,032][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:17:50,521][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:17:51,010][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:17:51,500][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:17:51,992][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:17:52,482][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:17:52,971][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:17:53,476][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:17:53,967][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:17:54,461][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:17:54,957][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:17:55,448][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:17:55,940][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:17:56,430][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:17:56,922][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:17:57,414][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:17:57,907][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:17:58,400][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:17:58,893][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:17:59,384][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:17:59,877][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:18:00,369][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:18:00,863][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:18:01,367][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:18:01,857][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:18:02,347][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:18:02,849][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:18:03,340][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:18:03,831][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:18:04,321][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:18:04,810][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:18:05,305][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:18:05,798][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:18:06,289][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:18:06,783][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:18:07,275][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:18:07,798][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:18:08,296][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:18:08,788][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:18:09,281][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:18:09,771][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:18:10,265][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:18:10,763][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:18:11,256][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:18:11,780][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:18:12,292][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:18:12,785][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9745 tokens.
[2025-11-12 22:18:13,473][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.99%, Current % of VRAM taken: 60.25%, Block Peak % of device VRAM: 61.75%, ΔTime: 00:00:32
[2025-11-12 22:18:14,220][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:18:14,221][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:18:14,223][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:18:15,423][__main__][INFO] - Iteration 7 took 48s (26.58% Gen, 70.97% Train). Generation: 12s, Training: 34s. Estimated remaining time: 40h 33m 24s. Estimated total time: 40h 41m 57s. Time estimates for 10 more iterations: 8m 8s, 100 more iterations: 1h 21m 23s, 500 more iterations: 6h 46m 59s.
[2025-11-12 22:18:15,425][__main__][INFO] - Starting iteration 7.
[2025-11-12 22:18:15,873][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:18:15,874][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:18:28,444][__main__][INFO] - Number of regex retries in iteration 7: 0
[2025-11-12 22:18:28,445][__main__][INFO] - agents played in iteration 7 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:18:29,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:18:29,317][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:18:29,343][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:18:29,366][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:18:29,366][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:18:29,367][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:18:30,020][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:18:30,488][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:18:30,983][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:18:31,474][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:18:31,965][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:18:32,455][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:18:32,947][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:18:33,437][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:18:33,927][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:18:34,422][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:18:34,913][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:18:35,407][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:18:35,901][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:18:36,416][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:18:36,916][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:18:37,407][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:18:37,901][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:18:38,393][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:18:38,883][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:18:39,387][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:18:39,879][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:18:40,370][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:18:40,864][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:18:41,355][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:18:41,846][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:18:42,342][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:18:42,835][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:18:43,327][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:18:43,823][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:18:44,315][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:18:44,809][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:18:45,302][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:18:45,793][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:18:46,286][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:18:46,777][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:18:47,276][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:18:47,768][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:18:48,262][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:18:48,762][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:18:49,254][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:18:49,745][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:18:50,237][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:18:50,726][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:18:51,217][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:18:51,708][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:18:52,201][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:18:52,702][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:18:53,192][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:18:53,683][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:18:54,178][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:18:54,669][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:18:55,162][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:18:55,654][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:18:56,146][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:18:56,643][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:18:57,132][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:18:57,621][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:18:58,111][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:18:58,603][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:18:59,096][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:18:59,587][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:19:00,080][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:19:00,585][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:19:01,075][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:19:01,566][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9646 tokens.
[2025-11-12 22:19:02,255][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 12.93%, Current % of VRAM taken: 59.19%, Block Peak % of device VRAM: 61.70%, ΔTime: 00:00:32
[2025-11-12 22:19:03,178][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:19:03,179][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:19:03,193][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:19:04,356][__main__][INFO] - Iteration 8 took 48s (25.93% Gen, 71.67% Train). Generation: 12s, Training: 34s. Estimated remaining time: 40h 14m 47s. Estimated total time: 40h 24m 10s. Time estimates for 10 more iterations: 8m 4s, 100 more iterations: 1h 20m 48s, 500 more iterations: 6h 44m 1s.
[2025-11-12 22:19:04,359][__main__][INFO] - Starting iteration 8.
[2025-11-12 22:19:04,799][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:19:04,800][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:19:05,482][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:19:17,296][__main__][INFO] - Number of regex retries in iteration 8: 1
[2025-11-12 22:19:17,297][__main__][INFO] - agents played in iteration 8 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:19:18,195][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 52.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:19:18,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 52.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:19:18,246][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 52.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:19:18,269][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 52.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:19:18,269][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:19:18,270][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:19:18,947][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:19:19,397][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:19:19,894][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:19:20,386][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:19:20,876][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:19:21,379][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:19:21,868][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:19:22,358][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:19:22,860][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:19:23,349][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:19:23,843][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:19:24,337][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:19:24,827][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:19:25,321][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:19:25,811][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:19:26,302][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:19:26,792][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:19:27,284][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:19:27,787][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:19:28,279][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:19:28,773][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:19:29,280][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:19:29,771][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:19:30,263][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:19:30,772][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:19:31,264][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:19:31,759][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:19:32,250][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:19:32,742][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:19:33,234][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:19:33,724][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:19:34,215][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:19:34,705][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:19:35,196][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:19:35,811][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:19:36,302][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:19:36,794][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:19:37,287][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:19:37,778][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:19:38,284][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:19:38,777][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:19:39,267][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:19:39,762][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:19:40,252][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:19:40,743][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:19:41,232][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:19:41,723][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:19:42,219][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:19:42,707][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:19:43,198][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:19:43,690][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:19:44,182][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:19:44,677][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:19:45,168][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:19:45,659][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:19:46,151][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:19:46,641][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:19:47,134][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:19:47,631][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:19:48,125][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:19:48,624][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:19:49,116][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:19:49,609][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:19:50,104][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:19:50,598][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9716 tokens.
[2025-11-12 22:19:51,285][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.00%, Current % of VRAM taken: 60.26%, Block Peak % of device VRAM: 61.68%, ΔTime: 00:00:32
[2025-11-12 22:19:52,024][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:19:52,025][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:19:52,027][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:19:52,960][__main__][INFO] - Iteration 9 took 48s (25.95% Gen, 72.11% Train). Generation: 12s, Training: 34s. Estimated remaining time: 39h 57m 50s. Estimated total time: 40h 8m 2s. Time estimates for 10 more iterations: 8m 1s, 100 more iterations: 1h 20m 16s, 500 more iterations: 6h 41m 20s.
[2025-11-12 22:19:52,962][__main__][INFO] - Starting iteration 9.
[2025-11-12 22:19:53,457][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:19:53,458][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:20:06,693][__main__][INFO] - Number of regex retries in iteration 9: 0
[2025-11-12 22:20:06,694][__main__][INFO] - agents played in iteration 9 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:20:07,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:20:07,747][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:20:07,774][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:20:07,797][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:20:07,798][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:20:07,799][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:20:08,481][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:20:08,936][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:20:09,436][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:20:09,927][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:20:10,422][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:20:10,915][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:20:11,407][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:20:11,899][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:20:12,396][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:20:12,889][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:20:13,400][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:20:13,893][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:20:14,387][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:20:14,885][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:20:15,378][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:20:15,868][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:20:16,361][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:20:16,852][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:20:17,345][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:20:17,835][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:20:18,327][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:20:18,815][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:20:19,304][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:20:19,806][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:20:20,295][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:20:20,785][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:20:21,288][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:20:21,777][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:20:22,266][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:20:22,756][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:20:23,245][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:20:23,738][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:20:24,229][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:20:24,717][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:20:25,210][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:20:25,700][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:20:26,190][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:20:26,680][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:20:27,170][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:20:27,663][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:20:28,153][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:20:28,644][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:20:29,136][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:20:29,626][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:20:30,114][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:20:30,604][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:20:31,093][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:20:31,596][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:20:32,085][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:20:32,575][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:20:33,076][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:20:33,564][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:20:34,057][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:20:34,548][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:20:35,040][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:20:35,537][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:20:36,027][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:20:36,519][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:20:37,012][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:20:37,505][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:20:38,010][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:20:38,504][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:20:38,995][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:20:39,488][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:20:39,981][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9681 tokens.
[2025-11-12 22:20:40,649][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.99%, Current % of VRAM taken: 60.25%, Block Peak % of device VRAM: 61.64%, ΔTime: 00:00:32
[2025-11-12 22:20:41,387][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:20:41,389][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:20:41,390][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:20:42,422][__main__][INFO] - Iteration 10 took 48s (27.03% Gen, 70.86% Train). Generation: 13s, Training: 34s. Estimated remaining time: 40h 37m 16s. Estimated total time: 40h 48m 17s. Time estimates for 10 more iterations: 8m 9s, 100 more iterations: 1h 21m 36s, 500 more iterations: 6h 48m 2s.
[2025-11-12 22:20:42,425][__main__][INFO] - Starting iteration 10.
[2025-11-12 22:20:42,905][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-12 22:20:42,906][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:20:56,473][__main__][INFO] - Number of regex retries in iteration 10: 0
[2025-11-12 22:20:56,474][__main__][INFO] - agents played in iteration 10 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:20:57,395][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:20:57,418][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:20:57,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:20:57,462][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:20:57,462][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:20:57,463][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:20:58,128][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:20:58,582][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:20:59,082][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:20:59,582][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:21:00,074][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:21:00,566][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:21:01,060][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:21:01,550][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:21:02,044][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:21:02,532][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:21:03,028][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:21:03,516][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:21:04,005][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:21:04,497][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:21:04,985][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:21:05,474][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:21:05,964][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:21:06,452][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:21:06,943][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:21:07,431][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:21:07,921][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:21:08,410][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:21:08,899][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:21:09,388][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:21:09,889][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:21:10,383][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:21:10,874][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:21:11,365][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:21:11,855][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:21:12,352][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:21:12,842][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:21:13,334][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:21:13,836][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:21:14,325][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:21:14,813][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:21:15,307][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:21:15,795][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:21:16,295][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:21:16,787][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:21:17,275][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:21:17,766][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:21:18,257][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:21:18,750][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:21:19,242][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:21:19,732][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:21:20,237][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:21:20,733][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:21:21,228][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:21:21,725][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:21:22,220][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:21:22,724][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:21:23,212][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:21:23,702][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:21:24,199][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:21:24,688][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:21:25,177][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:21:25,669][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:21:26,163][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:21:26,659][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:21:27,151][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:21:27,643][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:21:28,143][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:21:28,635][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:21:29,127][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:21:29,622][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9601 tokens.
[2025-11-12 22:21:30,326][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.93%, Current % of VRAM taken: 60.19%, Block Peak % of device VRAM: 61.58%, ΔTime: 00:00:32
[2025-11-12 22:21:31,097][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:21:31,099][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:21:31,101][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:21:32,976][__main__][INFO] - Iteration 11 took 50s (27.10% Gen, 69.15% Train). Generation: 13s, Training: 34s. Estimated remaining time: 41h 31m 43s. Estimated total time: 41h 43m 35s. Time estimates for 10 more iterations: 8m 20s, 100 more iterations: 1h 23m 27s, 500 more iterations: 6h 57m 15s.
[2025-11-12 22:21:32,978][__main__][INFO] - Starting iteration 11.
[2025-11-12 22:21:33,423][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:21:33,424][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:21:35,848][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:21:47,288][__main__][INFO] - Number of regex retries in iteration 11: 1
[2025-11-12 22:21:47,289][__main__][INFO] - agents played in iteration 11 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:21:48,115][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.24%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:21:48,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.24%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:21:48,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.24%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:21:48,191][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.24%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:21:48,192][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:21:48,192][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:21:48,799][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:21:49,260][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:21:49,760][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:21:50,252][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:21:50,746][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:21:51,242][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:21:51,742][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:21:52,235][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:21:52,726][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:21:53,218][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:21:53,709][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:21:54,202][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:21:54,694][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:21:55,186][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:21:55,682][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:21:56,175][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:21:56,667][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:21:57,159][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:21:57,649][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:21:58,143][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:21:58,638][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:21:59,133][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:21:59,639][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:22:00,131][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:22:00,622][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:22:01,128][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:22:01,619][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:22:02,110][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:22:02,601][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:22:03,092][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:22:03,584][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:22:04,076][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:22:04,567][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:22:05,062][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:22:05,552][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:22:06,044][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:22:06,534][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:22:07,026][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:22:07,538][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:22:08,030][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:22:08,521][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:22:09,015][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:22:09,506][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:22:09,998][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:22:10,493][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:22:10,985][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:22:11,480][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:22:11,970][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:22:12,464][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:22:12,957][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:22:13,449][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:22:13,947][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:22:14,440][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:22:14,931][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:22:15,422][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:22:15,915][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:22:16,406][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:22:16,905][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:22:17,397][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:22:17,894][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:22:18,387][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:22:18,880][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:22:19,376][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:22:19,868][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:22:20,365][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9650 tokens.
[2025-11-12 22:22:21,054][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.92%, Current % of VRAM taken: 60.18%, Block Peak % of device VRAM: 61.58%, ΔTime: 00:00:32
[2025-11-12 22:22:21,806][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:22:21,807][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:22:21,809][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:22:22,765][__main__][INFO] - Iteration 12 took 49s (28.10% Gen, 69.96% Train). Generation: 13s, Training: 34s. Estimated remaining time: 40h 54m 25s. Estimated total time: 41h 7m 6s. Time estimates for 10 more iterations: 8m 13s, 100 more iterations: 1h 22m 14s, 500 more iterations: 6h 51m 11s.
[2025-11-12 22:22:22,767][__main__][INFO] - Starting iteration 12.
[2025-11-12 22:22:23,223][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:22:23,224][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:22:24,403][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:22:32,901][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

This proposal is based on the per-item values given and the fact that both agents have different valuations for the items. Since I have a significant value for hats (10) compared to Bob's (1), it makes sense to propose keeping all the hats. Bob values books much higher than I do, so proposing to keep none of the books is a strategic choice. Similarly, while I value balls low and Bob values them higher, since we are equally likely to receive a proportional share if the total proposals exceed the item quantity, it is more beneficial to me to propose keeping none of the balls to avoid splitting them. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:22:38,365][__main__][INFO] - Number of regex retries in iteration 12: 2
[2025-11-12 22:22:38,366][__main__][INFO] - agents played in iteration 12 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:22:39,205][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.25%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:22:39,233][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.25%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:22:39,260][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.25%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:22:39,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.25%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:22:39,283][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:22:39,284][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:22:39,898][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:22:40,359][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:22:40,858][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:22:41,357][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:22:41,852][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:22:42,346][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:22:42,838][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:22:43,332][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:22:43,824][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:22:44,341][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:22:44,831][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:22:45,322][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:22:45,821][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:22:46,310][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:22:46,802][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:22:47,292][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:22:47,782][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:22:48,274][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:22:48,765][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:22:49,258][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:22:49,752][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:22:50,241][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:22:50,743][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:22:51,232][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:22:51,723][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:22:52,218][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:22:52,712][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:22:53,202][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:22:53,691][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:22:54,184][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:22:54,680][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:22:55,172][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:22:55,663][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:22:56,154][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:22:56,643][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:22:57,138][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:22:57,625][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:22:58,115][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:22:58,609][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:22:59,098][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:22:59,586][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:23:00,078][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:23:00,568][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:23:01,060][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:23:01,548][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:23:02,040][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:23:02,543][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:23:03,034][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:23:03,524][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:23:04,017][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:23:04,508][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:23:04,998][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:23:05,486][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:23:05,976][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:23:06,466][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:23:06,956][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:23:07,445][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:23:07,955][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:23:08,447][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:23:08,939][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:23:09,434][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:23:09,925][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:23:10,429][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:23:10,924][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:23:11,418][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9684 tokens.
[2025-11-12 22:23:12,091][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.94%, Current % of VRAM taken: 60.20%, Block Peak % of device VRAM: 61.66%, ΔTime: 00:00:32
[2025-11-12 22:23:12,837][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:23:12,839][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:23:12,840][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:23:13,819][__main__][INFO] - Iteration 13 took 50s (29.93% Gen, 68.14% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 56m 15s. Estimated total time: 42h 9m 48s. Time estimates for 10 more iterations: 8m 25s, 100 more iterations: 1h 24m 19s, 500 more iterations: 7h 1m 38s.
[2025-11-12 22:23:13,821][__main__][INFO] - Starting iteration 13.
[2025-11-12 22:23:14,286][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:23:14,287][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:23:22,412][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 10 books, 0 balls

Given the per-item values, it appears that Alice values hats more than I do, but I value books more than Alice does. Since I have a higher value for books and Alice has a higher value for hats, proposing to take all the books seems to be the best strategy. This way, I can maximize my points based on my high value for books. Hats and balls have equal values for both of us, but since I have a lower value for them, it's better to avoid those items entirely in this round. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:23:22,831][mllm.models.large_language_model_local][WARNING] - Response Given the current values, you should propose to take the hats while Alice will likely take the balls. Therefore, a strategic proposal would be:

Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:23:29,903][__main__][INFO] - Number of regex retries in iteration 13: 2
[2025-11-12 22:23:29,904][__main__][INFO] - agents played in iteration 13 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:23:30,840][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:23:30,868][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:23:30,894][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:23:30,917][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:23:30,918][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:23:30,919][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:23:31,562][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:23:32,016][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:23:32,514][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:23:33,012][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:23:33,505][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:23:33,997][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:23:34,492][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:23:34,984][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:23:35,477][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:23:35,969][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:23:36,462][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:23:36,969][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:23:37,463][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:23:37,955][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:23:38,447][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:23:38,942][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:23:39,457][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:23:39,948][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:23:40,441][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:23:40,938][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:23:41,429][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:23:41,919][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:23:42,408][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:23:42,898][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:23:43,390][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:23:43,882][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:23:44,379][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:23:44,872][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:23:45,363][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:23:45,862][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:23:46,358][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:23:46,850][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:23:47,361][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:23:47,853][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:23:48,344][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:23:48,834][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:23:49,326][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:23:49,837][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:23:50,328][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:23:50,819][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:23:51,324][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:23:51,815][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:23:52,305][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:23:52,796][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:23:53,287][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:23:53,779][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:23:54,270][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:23:54,765][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:23:55,257][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:23:55,748][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:23:56,251][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:23:56,742][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:23:57,232][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:23:57,740][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:23:58,228][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:23:58,719][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:23:59,210][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:23:59,702][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:24:00,196][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:24:00,691][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:24:01,187][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:24:01,686][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:24:02,184][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:24:02,679][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:24:03,174][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9705 tokens.
[2025-11-12 22:24:03,878][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.99%, Current % of VRAM taken: 60.25%, Block Peak % of device VRAM: 61.66%, ΔTime: 00:00:32
[2025-11-12 22:24:04,606][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:24:04,608][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:24:04,609][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:24:05,720][__main__][INFO] - Iteration 14 took 51s (30.36% Gen, 67.48% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 37m 18s. Estimated total time: 42h 51m 42s. Time estimates for 10 more iterations: 8m 34s, 100 more iterations: 1h 25m 43s, 500 more iterations: 7h 8m 37s.
[2025-11-12 22:24:05,722][__main__][INFO] - Starting iteration 14.
[2025-11-12 22:24:06,201][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:24:06,201][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:24:20,305][__main__][INFO] - Number of regex retries in iteration 14: 0
[2025-11-12 22:24:20,306][__main__][INFO] - agents played in iteration 14 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:24:21,217][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:24:21,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:24:21,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:24:21,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:24:21,295][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:24:21,296][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:24:21,942][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:24:22,396][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:24:22,894][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:24:23,390][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:24:23,886][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:24:24,379][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:24:24,870][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:24:25,375][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:24:25,866][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:24:26,358][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:24:26,852][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:24:27,344][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:24:27,863][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:24:28,352][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:24:28,843][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:24:29,341][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:24:29,833][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:24:30,323][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:24:30,813][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:24:31,304][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:24:31,799][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:24:32,289][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:24:32,778][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:24:33,269][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:24:33,759][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:24:34,248][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:24:34,737][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:24:35,226][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:24:35,716][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:24:36,204][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:24:36,693][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:24:37,184][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:24:37,674][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:24:38,166][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:24:38,659][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:24:39,150][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:24:39,657][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:24:40,157][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:24:40,647][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:24:41,153][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:24:41,645][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:24:42,137][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:24:42,643][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:24:43,135][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:24:43,630][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:24:44,122][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:24:44,612][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:24:45,105][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:24:45,600][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:24:46,091][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:24:46,581][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:24:47,071][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:24:47,561][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:24:48,052][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:24:48,542][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:24:49,036][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:24:49,525][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:24:50,016][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:24:50,510][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:24:51,005][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:24:51,500][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:24:51,994][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:24:52,486][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:24:52,988][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:24:53,485][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9561 tokens.
[2025-11-12 22:24:54,194][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.92%, Current % of VRAM taken: 60.18%, Block Peak % of device VRAM: 61.58%, ΔTime: 00:00:32
[2025-11-12 22:24:54,904][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:24:54,905][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:24:54,907][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:24:55,807][__main__][INFO] - Iteration 15 took 49s (28.43% Gen, 69.75% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 5m 7s. Estimated total time: 41h 20m 21s. Time estimates for 10 more iterations: 8m 16s, 100 more iterations: 1h 22m 40s, 500 more iterations: 6h 53m 23s.
[2025-11-12 22:24:55,809][__main__][INFO] - Starting iteration 15.
[2025-11-12 22:24:56,265][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:24:56,265][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:24:57,483][mllm.models.large_language_model_local][WARNING] - Response Proposals: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:25:01,203][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given Bob's per-item values, he values hats much lower than you do, so taking all the hats is the most beneficial for you. Since books and balls have similar values for both of you, splitting them would likely result in a lower overall point gain for you compared to taking all the hats. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:25:11,111][__main__][INFO] - Number of regex retries in iteration 15: 2
[2025-11-12 22:25:11,112][__main__][INFO] - agents played in iteration 15 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:25:11,965][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.22%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:25:11,991][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.22%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:25:12,016][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.22%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:25:12,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.22%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:25:12,039][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:25:12,041][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:25:12,683][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:25:13,134][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:25:13,648][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:25:14,152][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:25:14,651][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:25:15,161][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:25:15,653][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:25:16,146][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:25:16,649][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:25:17,141][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:25:17,632][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:25:18,123][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:25:18,614][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:25:19,107][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:25:19,598][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:25:20,090][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:25:20,581][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:25:21,080][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:25:21,577][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:25:22,079][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:25:22,573][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:25:23,068][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:25:23,564][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:25:24,056][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:25:24,550][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:25:25,043][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:25:25,549][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:25:26,043][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:25:26,535][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:25:27,027][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:25:27,519][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:25:28,011][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:25:28,504][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:25:28,997][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:25:29,490][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:25:29,983][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:25:30,477][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:25:30,981][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:25:31,473][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:25:31,966][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:25:32,462][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:25:32,953][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:25:33,459][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:25:33,950][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:25:34,442][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:25:34,946][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:25:35,437][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:25:35,928][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:25:36,418][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:25:36,912][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:25:37,408][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:25:37,899][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:25:38,390][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:25:38,888][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:25:39,379][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:25:39,870][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:25:40,362][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:25:40,852][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:25:41,360][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:25:41,854][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:25:42,349][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:25:42,854][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:25:43,352][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:25:43,851][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:25:44,349][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9757 tokens.
[2025-11-12 22:25:45,071][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.01%, ΔTime: 00:00:32
[2025-11-12 22:25:45,808][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:25:45,810][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:25:45,811][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:25:46,857][__main__][INFO] - Iteration 16 took 50s (29.35% Gen, 68.59% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 53m 33s. Estimated total time: 42h 9m 38s. Time estimates for 10 more iterations: 8m 25s, 100 more iterations: 1h 24m 19s, 500 more iterations: 7h 1m 36s.
[2025-11-12 22:25:46,859][__main__][INFO] - Starting iteration 16.
[2025-11-12 22:25:47,385][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:25:47,386][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:25:49,560][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:26:01,602][__main__][INFO] - Number of regex retries in iteration 16: 1
[2025-11-12 22:26:01,603][__main__][INFO] - agents played in iteration 16 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:26:02,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:26:02,461][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:26:02,487][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:26:02,510][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:26:02,510][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:26:02,511][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:26:03,102][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:26:03,561][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:26:04,058][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:26:04,551][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:26:05,042][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:26:05,536][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:26:06,031][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:26:06,523][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:26:07,015][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:26:07,507][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:26:07,999][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:26:08,491][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:26:08,982][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:26:09,474][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:26:09,983][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:26:10,473][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:26:10,965][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:26:11,463][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:26:11,955][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:26:12,445][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:26:12,935][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:26:13,426][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:26:13,924][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:26:14,416][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:26:14,908][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:26:15,402][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:26:15,893][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:26:16,385][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:26:16,878][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:26:17,368][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:26:17,876][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:26:18,369][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:26:18,863][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:26:19,366][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:26:19,858][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:26:20,351][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:26:20,844][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:26:21,338][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:26:21,834][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:26:22,328][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:26:22,821][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:26:23,313][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:26:23,806][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:26:24,299][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:26:24,791][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:26:25,285][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:26:25,797][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:26:26,293][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:26:26,787][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:26:27,283][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:26:27,776][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:26:28,281][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:26:28,774][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:26:29,270][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:26:29,766][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:26:30,262][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:26:30,756][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:26:31,248][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:26:31,742][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:26:32,235][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:26:32,731][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:26:33,230][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:26:33,726][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:26:34,223][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:26:34,720][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9693 tokens.
[2025-11-12 22:26:35,436][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.00%, Current % of VRAM taken: 60.26%, Block Peak % of device VRAM: 61.58%, ΔTime: 00:00:32
[2025-11-12 22:26:36,159][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:26:36,162][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:26:36,164][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:26:37,062][__main__][INFO] - Iteration 17 took 49s (28.62% Gen, 69.57% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 6m 56s. Estimated total time: 41h 23m 52s. Time estimates for 10 more iterations: 8m 16s, 100 more iterations: 1h 22m 47s, 500 more iterations: 6h 53m 58s.
[2025-11-12 22:26:37,065][__main__][INFO] - Starting iteration 17.
[2025-11-12 22:26:37,570][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:26:37,571][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:26:47,572][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given that the per-item values are significantly different between Alice and Bob, it's strategic to allocate items that maximize the value for the items where Alice has a higher value than Bob. In this case, Alice values hats and books more than Bob, and balls equally. By proposing to take all 10 hats, we aim to secure the maximum points from the items where we have a significant advantage. Since the allocation is proportional and Bob values books and balls more, taking all the hats ensures we get the highest possible value from the hats, which is 10. We can then move on to splitting books and balls in future rounds if any are left. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:26:52,876][__main__][INFO] - Number of regex retries in iteration 17: 1
[2025-11-12 22:26:52,877][__main__][INFO] - agents played in iteration 17 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:26:53,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:26:53,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:26:53,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:26:53,762][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:26:53,762][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:26:53,763][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:26:54,363][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:26:54,819][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:26:55,313][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:26:55,914][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:26:56,407][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:26:56,897][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:26:57,401][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:26:57,893][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:26:58,384][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:26:58,883][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:26:59,377][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:26:59,880][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:27:00,374][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:27:00,869][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:27:01,363][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:27:01,856][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:27:02,350][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:27:02,844][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:27:03,335][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:27:03,827][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:27:04,319][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:27:04,810][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:27:05,309][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:27:05,803][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:27:06,296][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:27:06,787][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:27:07,279][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:27:07,776][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:27:08,268][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:27:08,760][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:27:09,253][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:27:09,746][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:27:10,238][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:27:10,733][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:27:11,222][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:27:11,714][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:27:12,205][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:27:12,695][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:27:13,188][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:27:13,679][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:27:14,173][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:27:14,667][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:27:15,160][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:27:15,654][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:27:16,146][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:27:16,639][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:27:17,132][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:27:17,623][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:27:18,114][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:27:18,607][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:27:19,100][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:27:19,591][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:27:20,086][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:27:20,580][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:27:21,086][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:27:21,577][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:27:22,071][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:27:22,579][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:27:23,077][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:27:23,575][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:27:24,073][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:27:24,571][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:27:25,070][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:27:25,568][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:27:26,093][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9780 tokens.
[2025-11-12 22:27:26,807][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 61.89%, ΔTime: 00:00:32
[2025-11-12 22:27:27,556][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:27:27,558][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:27:27,560][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:27:28,459][__main__][INFO] - Iteration 18 took 50s (30.08% Gen, 68.15% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 6m 41s. Estimated total time: 42h 24m 27s. Time estimates for 10 more iterations: 8m 28s, 100 more iterations: 1h 24m 48s, 500 more iterations: 7h 4m 4s.
[2025-11-12 22:27:28,462][__main__][INFO] - Starting iteration 18.
[2025-11-12 22:27:28,934][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:27:28,934][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:27:43,074][__main__][INFO] - Number of regex retries in iteration 18: 0
[2025-11-12 22:27:43,075][__main__][INFO] - agents played in iteration 18 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:27:43,893][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:27:43,917][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:27:43,942][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:27:43,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:27:43,965][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:27:43,966][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:27:44,574][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:27:45,027][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:27:45,527][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:27:46,021][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:27:46,511][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:27:47,005][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:27:47,510][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:27:48,002][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:27:48,493][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:27:48,987][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:27:49,476][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:27:49,966][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:27:50,459][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:27:50,955][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:27:51,447][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:27:51,938][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:27:52,430][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:27:52,923][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:27:53,418][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:27:53,919][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:27:54,414][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:27:54,906][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:27:55,402][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:27:55,894][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:27:56,386][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:27:56,883][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:27:57,373][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:27:57,866][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:27:58,360][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:27:58,851][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:27:59,347][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:27:59,841][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:28:00,332][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:28:00,824][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:28:01,319][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:28:01,817][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:28:02,311][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:28:02,802][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:28:03,297][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:28:03,787][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:28:04,279][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:28:04,770][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:28:05,265][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:28:05,756][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:28:06,249][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:28:06,739][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:28:07,231][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:28:07,724][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:28:08,214][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:28:08,706][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:28:09,199][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:28:09,705][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:28:10,198][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:28:10,688][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:28:11,183][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:28:11,674][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:28:12,166][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:28:12,659][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:28:13,151][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:28:13,647][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:28:14,142][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:28:14,637][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:28:15,134][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:28:15,629][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:28:16,158][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9678 tokens.
[2025-11-12 22:28:16,856][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 61.61%, ΔTime: 00:00:32
[2025-11-12 22:28:17,623][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:28:17,624][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:28:17,626][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:28:18,900][__main__][INFO] - Iteration 19 took 49s (28.30% Gen, 69.15% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 19m 45s. Estimated total time: 41h 38m 22s. Time estimates for 10 more iterations: 8m 19s, 100 more iterations: 1h 23m 16s, 500 more iterations: 6h 56m 23s.
[2025-11-12 22:28:18,903][__main__][INFO] - Starting iteration 19.
[2025-11-12 22:28:19,379][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:28:19,379][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:28:20,861][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:28:29,621][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:28:30,644][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 20 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:28:34,036][__main__][INFO] - Number of regex retries in iteration 19: 3
[2025-11-12 22:28:34,037][__main__][INFO] - agents played in iteration 19 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:28:34,843][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:28:34,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:28:34,892][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:28:34,914][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:28:34,914][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:28:34,915][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:28:35,508][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:28:35,968][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:28:36,468][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:28:36,967][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:28:37,461][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:28:37,954][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:28:38,451][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:28:38,944][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:28:39,437][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:28:39,932][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:28:40,423][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:28:40,914][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:28:41,407][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:28:41,898][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:28:42,408][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:28:42,899][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:28:43,391][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:28:43,890][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:28:44,380][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:28:44,871][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:28:45,361][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:28:45,851][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:28:46,354][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:28:46,854][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:28:47,351][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:28:47,853][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:28:48,353][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:28:48,848][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:28:49,344][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:28:49,838][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:28:50,331][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:28:50,823][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:28:51,319][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:28:51,810][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:28:52,303][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:28:52,812][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:28:53,304][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:28:53,796][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:28:54,292][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:28:54,785][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:28:55,283][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:28:55,774][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:28:56,265][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:28:56,756][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:28:57,247][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:28:57,739][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:28:58,233][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:28:58,722][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:28:59,214][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:28:59,706][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:29:00,201][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:29:00,696][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:29:01,189][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:29:01,684][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:29:02,177][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:29:02,673][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:29:03,168][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:29:03,669][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:29:04,164][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:29:04,662][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:29:05,155][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:29:05,650][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:29:06,149][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:29:06,647][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:29:07,144][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9739 tokens.
[2025-11-12 22:29:07,850][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.99%, Current % of VRAM taken: 60.25%, Block Peak % of device VRAM: 61.84%, ΔTime: 00:00:32
[2025-11-12 22:29:08,642][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:29:08,644][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:29:08,645][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:29:09,595][__main__][INFO] - Iteration 20 took 50s (29.19% Gen, 68.92% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 31m 24s. Estimated total time: 41h 50m 52s. Time estimates for 10 more iterations: 8m 22s, 100 more iterations: 1h 23m 41s, 500 more iterations: 6h 58m 28s.
[2025-11-12 22:29:09,603][__main__][INFO] - Starting iteration 20.
[2025-11-12 22:29:10,078][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-12 22:29:10,079][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:29:12,461][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:29:19,416][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given Alice's per-item values, she will likely propose to take most of the books and balls, leaving me with the hats. By proposing to take all 10 hats, I am maximizing my share of the items I value the most, which are hats. Alice will likely propose to take all the books and balls since those are her highest-value items. If the sum of our proposals exceeds the item quantity, the items will be allocated proportionally, but in this case, my proposal does not exceed the item quantity, so I will receive all 10 hats. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:29:25,322][__main__][INFO] - Number of regex retries in iteration 20: 2
[2025-11-12 22:29:25,323][__main__][INFO] - agents played in iteration 20 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:29:26,124][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:29:26,152][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:29:26,178][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:29:26,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:29:26,201][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:29:26,201][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:29:26,792][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:29:27,248][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:29:27,743][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:29:28,240][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:29:28,733][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:29:29,227][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:29:29,727][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:29:30,217][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:29:30,713][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:29:31,203][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:29:31,694][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:29:32,189][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:29:32,679][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:29:33,174][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:29:33,668][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:29:34,157][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:29:34,648][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:29:35,139][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:29:35,631][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:29:36,127][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:29:36,618][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:29:37,112][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:29:37,606][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:29:38,098][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:29:38,589][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:29:39,092][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:29:39,584][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:29:40,082][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:29:40,577][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:29:41,070][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:29:41,562][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:29:42,052][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:29:42,543][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:29:43,033][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:29:43,524][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:29:44,024][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:29:44,516][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:29:45,011][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:29:45,520][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:29:46,010][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:29:46,501][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:29:46,992][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:29:47,483][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:29:47,978][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:29:48,469][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:29:48,961][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:29:49,452][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:29:49,945][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:29:50,437][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:29:50,927][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:29:51,419][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:29:51,922][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:29:52,414][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:29:52,904][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:29:53,399][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:29:53,890][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:29:54,382][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:29:54,876][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:29:55,370][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:29:55,868][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:29:56,366][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:29:56,863][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:29:57,362][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:29:57,859][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:29:58,355][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9768 tokens.
[2025-11-12 22:29:59,080][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 61.69%, ΔTime: 00:00:32
[2025-11-12 22:29:59,918][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:29:59,926][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:29:59,939][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:30:02,208][__main__][INFO] - Iteration 21 took 52s (29.24% Gen, 66.40% Train). Generation: 15s, Training: 34s. Estimated remaining time: 43h 6m 12s. Estimated total time: 43h 26m 32s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 53s, 500 more iterations: 7h 14m 25s.
[2025-11-12 22:30:02,210][__main__][INFO] - Starting iteration 21.
[2025-11-12 22:30:02,689][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:30:02,689][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:30:13,484][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:30:17,156][__main__][INFO] - Number of regex retries in iteration 21: 1
[2025-11-12 22:30:17,157][__main__][INFO] - agents played in iteration 21 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:30:18,013][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:30:18,036][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:30:18,060][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:30:18,082][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:30:18,082][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:30:18,083][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:30:18,690][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:30:19,141][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:30:19,640][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:30:20,135][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:30:20,635][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:30:21,127][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:30:21,620][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:30:22,112][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:30:22,604][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:30:23,145][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:30:23,642][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:30:24,136][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:30:24,629][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:30:25,123][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:30:25,627][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:30:26,119][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:30:26,610][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:30:27,116][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:30:27,608][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:30:28,099][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:30:28,603][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:30:29,095][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:30:29,588][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:30:30,080][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:30:30,570][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:30:31,066][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:30:31,560][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:30:32,052][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:30:32,544][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:30:33,035][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:30:33,533][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:30:34,026][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:30:34,519][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:30:35,014][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:30:35,504][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:30:35,993][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:30:36,484][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:30:36,974][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:30:37,468][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:30:37,958][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:30:38,448][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:30:38,945][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:30:39,436][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:30:39,926][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:30:40,421][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:30:40,911][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:30:41,408][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:30:41,904][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:30:42,395][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:30:42,888][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:30:43,379][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:30:43,875][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:30:44,369][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:30:44,862][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:30:45,358][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:30:45,848][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:30:46,342][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:30:46,843][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:30:47,338][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:30:47,832][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:30:48,342][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:30:48,836][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:30:49,332][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:30:49,828][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:30:50,329][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9800 tokens.
[2025-11-12 22:30:51,072][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 61.79%, ΔTime: 00:00:32
[2025-11-12 22:30:51,870][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:30:51,871][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:30:51,874][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:30:52,916][__main__][INFO] - Iteration 22 took 50s (28.80% Gen, 69.12% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 30m 10s. Estimated total time: 41h 51m 22s. Time estimates for 10 more iterations: 8m 22s, 100 more iterations: 1h 23m 42s, 500 more iterations: 6h 58m 33s.
[2025-11-12 22:30:52,918][__main__][INFO] - Starting iteration 22.
[2025-11-12 22:30:53,391][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:30:53,391][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:31:07,807][__main__][INFO] - Number of regex retries in iteration 22: 0
[2025-11-12 22:31:07,808][__main__][INFO] - agents played in iteration 22 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:31:08,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:31:08,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:31:08,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:31:08,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:31:08,719][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:31:08,720][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:31:09,318][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:31:09,768][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:31:10,268][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:31:10,761][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:31:11,252][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:31:11,745][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:31:12,235][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:31:12,746][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:31:13,236][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:31:13,731][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:31:14,227][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:31:14,721][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:31:15,214][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:31:15,714][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:31:16,206][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:31:16,718][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:31:17,209][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:31:17,701][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:31:18,192][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:31:18,692][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:31:19,185][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:31:19,676][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:31:20,168][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:31:20,663][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:31:21,154][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:31:21,646][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:31:22,136][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:31:22,627][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:31:23,124][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:31:23,614][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:31:24,105][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:31:24,595][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:31:25,086][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:31:25,579][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:31:26,072][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:31:26,566][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:31:27,061][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:31:27,552][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:31:28,044][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:31:28,538][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:31:29,034][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:31:29,528][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:31:30,020][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:31:30,510][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:31:31,003][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:31:31,501][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:31:31,993][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:31:32,486][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:31:32,977][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:31:33,468][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:31:33,962][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:31:34,454][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:31:34,945][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:31:35,438][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:31:35,930][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:31:36,432][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:31:36,926][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:31:37,420][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:31:37,929][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:31:38,424][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:31:38,917][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:31:39,412][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:31:39,909][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:31:40,403][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:31:40,898][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9745 tokens.
[2025-11-12 22:31:41,606][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.98%, Current % of VRAM taken: 60.24%, Block Peak % of device VRAM: 61.85%, ΔTime: 00:00:32
[2025-11-12 22:31:42,410][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:31:42,412][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:31:42,413][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:31:43,413][__main__][INFO] - Iteration 23 took 50s (28.82% Gen, 69.18% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 19m 6s. Estimated total time: 41h 41m 8s. Time estimates for 10 more iterations: 8m 20s, 100 more iterations: 1h 23m 22s, 500 more iterations: 6h 56m 51s.
[2025-11-12 22:31:43,415][__main__][INFO] - Starting iteration 23.
[2025-11-12 22:31:43,897][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:31:43,898][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:31:57,903][__main__][INFO] - Number of regex retries in iteration 23: 0
[2025-11-12 22:31:57,903][__main__][INFO] - agents played in iteration 23 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:31:58,793][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:31:58,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:31:58,847][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:31:58,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:31:58,884][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:31:58,885][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:31:59,509][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:31:59,956][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:32:00,456][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:32:00,950][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:32:01,441][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:32:01,931][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:32:02,421][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:32:02,912][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:32:03,407][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:32:03,897][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:32:04,427][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:32:04,920][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:32:05,411][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:32:05,903][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:32:06,393][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:32:06,887][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:32:07,381][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:32:07,872][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:32:08,367][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:32:08,856][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:32:09,346][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:32:09,840][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:32:10,331][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:32:10,822][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:32:11,312][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:32:11,802][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:32:12,294][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:32:12,785][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:32:13,276][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:32:13,769][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:32:14,262][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:32:14,753][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:32:15,247][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:32:15,737][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:32:16,247][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:32:16,739][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:32:17,232][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:32:17,726][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:32:18,217][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:32:18,709][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:32:19,199][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:32:19,689][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:32:20,184][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:32:20,675][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:32:21,166][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:32:21,657][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:32:22,147][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:32:22,637][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:32:23,126][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:32:23,617][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:32:24,122][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:32:24,612][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:32:25,103][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:32:25,603][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:32:26,093][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:32:26,586][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:32:27,083][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:32:27,574][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:32:28,069][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:32:28,565][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:32:29,060][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:32:29,556][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:32:30,051][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:32:30,549][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:32:31,042][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9785 tokens.
[2025-11-12 22:32:31,764][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.00%, Current % of VRAM taken: 60.26%, Block Peak % of device VRAM: 61.61%, ΔTime: 00:00:32
[2025-11-12 22:32:32,537][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:32:32,539][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:32:32,541][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:32:33,570][__main__][INFO] - Iteration 24 took 49s (28.19% Gen, 69.73% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 0m 49s. Estimated total time: 41h 23m 41s. Time estimates for 10 more iterations: 8m 16s, 100 more iterations: 1h 22m 47s, 500 more iterations: 6h 53m 56s.
[2025-11-12 22:32:33,572][__main__][INFO] - Starting iteration 24.
[2025-11-12 22:32:34,136][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:32:34,137][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:32:42,186][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:32:47,946][__main__][INFO] - Number of regex retries in iteration 24: 1
[2025-11-12 22:32:47,947][__main__][INFO] - agents played in iteration 24 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:32:48,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:32:48,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:32:48,860][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:32:48,882][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:32:48,882][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:32:48,883][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:32:49,518][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:32:49,971][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:32:50,474][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:32:50,969][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:32:51,464][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:32:51,957][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:32:52,451][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:32:52,964][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:32:53,459][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:32:53,952][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:32:54,442][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:32:54,933][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:32:55,424][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:32:55,914][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:32:56,405][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:32:56,894][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:32:57,384][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:32:57,874][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:32:58,374][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:32:58,870][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:32:59,367][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:32:59,858][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:33:00,350][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:33:00,858][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:33:01,350][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:33:01,841][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:33:02,331][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:33:02,826][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:33:03,321][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:33:03,811][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:33:04,301][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:33:04,794][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:33:05,287][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:33:05,779][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:33:06,273][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:33:06,767][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:33:07,274][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:33:07,767][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:33:08,260][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:33:08,756][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:33:09,246][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:33:09,742][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:33:10,233][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:33:10,725][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:33:11,222][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:33:11,713][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:33:12,204][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:33:12,697][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:33:13,188][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:33:13,682][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:33:14,171][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:33:14,663][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:33:15,154][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:33:15,644][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:33:16,134][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:33:16,642][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:33:17,135][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:33:17,635][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:33:18,130][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:33:18,625][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:33:19,123][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:33:19,614][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:33:20,109][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:33:20,602][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:33:21,094][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9736 tokens.
[2025-11-12 22:33:21,744][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.95%, Current % of VRAM taken: 60.21%, Block Peak % of device VRAM: 61.54%, ΔTime: 00:00:32
[2025-11-12 22:33:22,523][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:33:22,524][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:33:22,526][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:33:23,564][__main__][INFO] - Iteration 25 took 49s (27.94% Gen, 69.96% Train). Generation: 13s, Training: 34s. Estimated remaining time: 40h 47m 40s. Estimated total time: 41h 11m 22s. Time estimates for 10 more iterations: 8m 14s, 100 more iterations: 1h 22m 22s, 500 more iterations: 6h 51m 53s.
[2025-11-12 22:33:23,566][__main__][INFO] - Starting iteration 25.
[2025-11-12 22:33:24,055][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:33:24,056][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:33:28,597][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:33:38,007][__main__][INFO] - Number of regex retries in iteration 25: 1
[2025-11-12 22:33:38,007][__main__][INFO] - agents played in iteration 25 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:33:38,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:33:38,881][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:33:38,905][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:33:38,928][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:33:38,929][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:33:38,930][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:33:39,579][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:33:40,042][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:33:40,547][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:33:41,041][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:33:41,547][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:33:42,040][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:33:42,533][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:33:43,029][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:33:43,528][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:33:44,024][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:33:44,519][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:33:45,011][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:33:45,506][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:33:46,000][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:33:46,495][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:33:46,989][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:33:47,481][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:33:47,984][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:33:48,477][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:33:48,968][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:33:49,462][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:33:49,952][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:33:50,445][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:33:50,939][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:33:51,430][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:33:51,924][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:33:52,419][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:33:52,912][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:33:53,406][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:33:53,899][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:33:54,393][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:33:54,886][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:33:55,378][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:33:55,883][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:33:56,375][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:33:56,866][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:33:57,359][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:33:57,850][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:33:58,340][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:33:58,829][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:33:59,319][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:33:59,809][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:34:00,300][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:34:00,790][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:34:01,282][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:34:01,772][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:34:02,264][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:34:02,755][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:34:03,247][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:34:03,742][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:34:04,232][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:34:04,724][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:34:05,222][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:34:05,714][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:34:06,209][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:34:06,704][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:34:07,198][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:34:07,697][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:34:08,193][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:34:08,686][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:34:09,180][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:34:09,674][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:34:10,167][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:34:10,661][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:34:11,154][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9774 tokens.
[2025-11-12 22:34:11,800][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.95%, Current % of VRAM taken: 60.21%, Block Peak % of device VRAM: 61.72%, ΔTime: 00:00:32
[2025-11-12 22:34:12,565][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:34:12,567][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:34:12,571][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:34:13,544][__main__][INFO] - Iteration 26 took 49s (28.19% Gen, 69.84% Train). Generation: 13s, Training: 34s. Estimated remaining time: 40h 49m 57s. Estimated total time: 41h 14m 28s. Time estimates for 10 more iterations: 8m 14s, 100 more iterations: 1h 22m 28s, 500 more iterations: 6h 52m 24s.
[2025-11-12 22:34:13,547][__main__][INFO] - Starting iteration 26.
[2025-11-12 22:34:14,029][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:34:14,030][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:34:16,319][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:34:28,340][__main__][INFO] - Number of regex retries in iteration 26: 1
[2025-11-12 22:34:28,341][__main__][INFO] - agents played in iteration 26 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:34:29,210][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:34:29,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:34:29,266][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:34:29,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:34:29,291][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:34:29,291][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:34:29,949][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:34:30,401][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:34:30,900][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:34:31,403][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:34:31,897][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:34:32,389][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:34:32,885][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:34:33,377][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:34:33,884][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:34:34,377][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:34:34,871][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:34:35,385][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:34:35,879][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:34:36,372][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:34:36,872][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:34:37,365][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:34:37,879][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:34:38,372][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:34:38,866][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:34:39,368][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:34:39,864][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:34:40,356][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:34:40,849][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:34:41,344][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:34:41,840][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:34:42,332][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:34:42,825][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:34:43,316][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:34:43,809][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:34:44,305][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:34:44,797][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:34:45,288][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:34:45,785][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:34:46,280][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:34:46,773][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:34:47,267][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:34:47,760][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:34:48,258][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:34:48,750][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:34:49,243][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:34:49,737][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:34:50,229][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:34:50,721][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:34:51,212][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:34:51,704][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:34:52,227][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:34:52,719][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:34:53,211][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:34:53,704][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:34:54,196][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:34:54,688][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:34:55,180][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:34:55,673][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:34:56,183][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:34:56,678][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:34:57,172][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:34:57,677][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:34:58,172][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:34:58,672][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:34:59,167][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:34:59,664][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:35:00,174][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:35:00,671][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:35:01,168][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:35:01,666][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9839 tokens.
[2025-11-12 22:35:02,320][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 61.81%, ΔTime: 00:00:32
[2025-11-12 22:35:03,091][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:35:03,093][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:35:03,094][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:35:04,147][__main__][INFO] - Iteration 27 took 50s (28.55% Gen, 69.34% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 20m 33s. Estimated total time: 41h 45m 55s. Time estimates for 10 more iterations: 8m 21s, 100 more iterations: 1h 23m 31s, 500 more iterations: 6h 57m 39s.
[2025-11-12 22:35:04,149][__main__][INFO] - Starting iteration 27.
[2025-11-12 22:35:04,619][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:35:04,619][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:35:05,760][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:35:07,457][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 5 hats, 5 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:35:10,517][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given the values, I (Bob) have a very low value for hats (1), while Alice values them highly (10). It makes sense to keep all the hats to avoid any potential loss. I will keep the hats as my high-value items for Alice are balls and books, so I propose to keep all 10 hats and none of the books or balls. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:35:19,603][__main__][INFO] - Number of regex retries in iteration 27: 3
[2025-11-12 22:35:19,604][__main__][INFO] - agents played in iteration 27 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:35:20,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:35:20,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:35:20,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:35:20,634][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:35:20,635][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:35:20,636][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:35:21,304][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:35:21,757][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:35:22,262][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:35:22,766][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:35:23,262][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:35:23,768][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:35:24,264][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:35:24,760][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:35:25,255][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:35:25,749][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:35:26,246][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:35:26,740][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:35:27,235][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:35:27,729][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:35:28,225][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:35:28,726][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:35:29,224][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:35:29,719][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:35:30,213][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:35:30,705][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:35:31,199][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:35:31,690][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:35:32,181][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:35:32,695][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:35:33,185][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:35:33,678][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:35:34,179][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:35:34,673][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:35:35,167][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:35:35,658][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:35:36,150][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:35:36,646][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:35:37,140][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:35:37,632][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:35:38,126][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:35:38,619][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:35:39,112][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:35:39,605][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:35:40,098][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:35:40,609][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:35:41,102][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:35:41,592][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:35:42,090][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:35:42,583][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:35:43,075][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:35:43,567][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:35:44,060][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:35:44,559][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:35:45,054][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:35:45,548][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:35:46,046][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:35:46,540][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:35:47,034][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:35:47,533][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:35:48,029][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:35:48,526][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:35:49,023][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:35:49,515][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:35:50,010][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:35:50,505][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:35:50,995][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:35:51,485][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:35:51,977][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:35:52,469][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:35:52,961][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9828 tokens.
[2025-11-12 22:35:53,656][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.93%, Current % of VRAM taken: 60.19%, Block Peak % of device VRAM: 61.95%, ΔTime: 00:00:32
[2025-11-12 22:35:54,415][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:35:54,416][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:35:54,418][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:35:55,439][__main__][INFO] - Iteration 28 took 50s (29.49% Gen, 68.50% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 54m 50s. Estimated total time: 42h 21m 4s. Time estimates for 10 more iterations: 8m 28s, 100 more iterations: 1h 24m 42s, 500 more iterations: 7h 3m 30s.
[2025-11-12 22:35:55,442][__main__][INFO] - Starting iteration 28.
[2025-11-12 22:35:56,097][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:35:56,098][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:36:10,388][__main__][INFO] - Number of regex retries in iteration 28: 0
[2025-11-12 22:36:10,389][__main__][INFO] - agents played in iteration 28 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:36:11,239][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:36:11,262][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:36:11,284][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:36:11,305][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:36:11,306][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:36:11,307][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:36:11,978][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:36:12,432][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:36:12,938][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:36:13,435][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:36:13,929][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:36:14,429][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:36:14,922][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:36:15,416][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:36:15,923][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:36:16,415][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:36:16,908][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:36:17,404][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:36:17,896][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:36:18,390][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:36:18,891][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:36:19,395][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:36:19,902][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:36:20,394][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:36:20,890][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:36:21,388][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:36:21,880][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:36:22,378][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:36:22,870][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:36:23,363][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:36:23,856][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:36:24,349][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:36:24,845][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:36:25,335][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:36:25,827][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:36:26,326][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:36:26,818][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:36:27,309][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:36:27,804][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:36:28,295][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:36:28,787][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:36:29,280][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:36:29,772][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:36:30,267][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:36:30,759][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:36:31,251][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:36:31,745][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:36:32,244][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:36:32,760][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:36:33,260][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:36:33,753][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:36:34,253][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:36:34,753][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:36:35,249][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:36:35,750][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:36:36,246][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:36:36,739][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:36:37,234][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:36:37,725][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:36:38,225][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:36:38,721][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:36:39,214][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:36:39,710][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:36:40,207][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:36:40,703][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:36:41,202][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:36:41,697][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:36:42,207][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:36:42,701][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:36:43,196][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:36:43,695][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9884 tokens.
[2025-11-12 22:36:44,329][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 61.97%, ΔTime: 00:00:32
[2025-11-12 22:36:45,095][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:36:45,098][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:36:45,101][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:36:46,206][__main__][INFO] - Iteration 29 took 50s (28.52% Gen, 69.27% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 18m 23s. Estimated total time: 41h 45m 28s. Time estimates for 10 more iterations: 8m 21s, 100 more iterations: 1h 23m 30s, 500 more iterations: 6h 57m 34s.
[2025-11-12 22:36:46,208][__main__][INFO] - Starting iteration 29.
[2025-11-12 22:36:46,674][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:36:46,675][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:36:47,795][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:36:47,871][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:37:00,981][__main__][INFO] - Number of regex retries in iteration 29: 2
[2025-11-12 22:37:00,981][__main__][INFO] - agents played in iteration 29 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:37:01,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:37:01,854][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:37:01,879][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:37:01,902][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:37:01,902][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:37:01,903][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:37:02,552][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:37:03,013][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:37:03,514][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:37:04,010][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:37:04,522][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:37:05,018][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:37:05,514][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:37:06,011][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:37:06,504][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:37:07,009][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:37:07,501][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:37:07,994][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:37:08,502][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:37:08,994][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:37:09,487][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:37:09,981][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:37:10,472][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:37:10,966][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:37:11,458][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:37:11,950][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:37:12,446][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:37:12,937][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:37:13,431][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:37:13,922][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:37:14,419][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:37:14,920][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:37:15,410][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:37:15,906][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:37:16,414][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:37:16,905][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:37:17,395][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:37:17,894][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:37:18,383][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:37:18,877][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:37:19,367][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:37:19,857][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:37:20,347][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:37:20,837][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:37:21,326][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:37:21,816][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:37:22,307][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:37:22,814][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:37:23,307][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:37:23,799][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:37:24,305][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:37:24,800][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:37:25,297][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:37:25,794][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:37:26,291][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:37:26,787][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:37:27,277][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:37:27,770][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:37:28,263][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:37:28,758][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:37:29,253][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:37:29,750][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:37:30,245][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:37:30,744][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:37:31,235][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:37:31,734][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:37:32,232][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:37:32,723][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:37:33,227][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:37:33,717][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:37:34,208][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9800 tokens.
[2025-11-12 22:37:34,866][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.91%, Current % of VRAM taken: 60.17%, Block Peak % of device VRAM: 61.69%, ΔTime: 00:00:32
[2025-11-12 22:37:35,615][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:37:35,617][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:37:35,619][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:37:36,582][__main__][INFO] - Iteration 30 took 49s (28.67% Gen, 69.40% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 7m 31s. Estimated total time: 41h 35m 26s. Time estimates for 10 more iterations: 8m 19s, 100 more iterations: 1h 23m 10s, 500 more iterations: 6h 55m 54s.
[2025-11-12 22:37:36,585][__main__][INFO] - Starting iteration 30.
[2025-11-12 22:37:37,068][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-12 22:37:37,069][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:37:50,986][__main__][INFO] - Number of regex retries in iteration 30: 0
[2025-11-12 22:37:50,987][__main__][INFO] - agents played in iteration 30 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:37:51,825][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.23%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:37:51,852][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.23%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:37:51,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.23%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:37:51,902][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.23%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:37:51,903][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:37:51,904][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:37:52,500][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:37:52,948][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:37:53,445][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:37:53,935][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:37:54,434][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:37:54,927][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:37:55,420][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:37:55,913][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:37:56,403][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:37:56,898][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:37:57,389][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:37:57,881][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:37:58,372][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:37:58,865][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:37:59,359][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:37:59,858][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:38:00,349][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:38:00,841][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:38:01,331][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:38:01,822][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:38:02,321][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:38:02,811][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:38:03,303][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:38:03,815][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:38:04,312][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:38:04,804][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:38:05,311][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:38:05,803][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:38:06,307][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:38:06,799][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:38:07,292][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:38:07,792][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:38:08,284][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:38:08,777][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:38:09,266][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:38:09,759][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:38:10,253][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:38:10,746][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:38:11,237][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:38:11,730][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:38:12,221][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:38:12,722][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:38:13,219][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:38:13,713][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:38:14,234][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:38:14,730][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:38:15,228][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:38:15,724][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:38:16,223][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:38:16,741][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:38:17,241][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:38:17,736][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:38:18,238][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:38:18,735][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:38:19,257][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:38:19,752][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:38:20,255][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:38:20,754][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:38:21,248][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:38:21,740][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:38:22,236][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:38:22,731][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:38:23,234][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:38:23,726][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:38:24,223][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9686 tokens.
[2025-11-12 22:38:24,878][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 61.92%, ΔTime: 00:00:32
[2025-11-12 22:38:25,768][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:38:25,770][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:38:25,772][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:38:27,679][__main__][INFO] - Iteration 31 took 50s (27.50% Gen, 68.73% Train). Generation: 13s, Training: 34s. Estimated remaining time: 41h 41m 47s. Estimated total time: 42h 10m 33s. Time estimates for 10 more iterations: 8m 26s, 100 more iterations: 1h 24m 21s, 500 more iterations: 7h 1m 45s.
[2025-11-12 22:38:27,681][__main__][INFO] - Starting iteration 31.
[2025-11-12 22:38:28,152][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:38:28,153][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:38:38,663][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given the values:
- Hats: You value at 10, Bob at 1
- Books: You value at 10, Bob at 10
- Balls: You value at 1, Bob at 10

Since you have a much higher valuation for hats compared to Bob, and you also value books moderately higher than Bob but balls much less, it's logical to prioritize taking all the hats. This ensures you get the full value for each hat you take. Books and balls are less favorable, so it's better to propose taking none of them to maximize your points from the hats. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:38:44,057][__main__][INFO] - Number of regex retries in iteration 31: 1
[2025-11-12 22:38:44,057][__main__][INFO] - agents played in iteration 31 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:38:44,831][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:38:44,859][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:38:44,888][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:38:44,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:38:44,913][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:38:44,914][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:38:45,511][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:38:45,966][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:38:46,464][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:38:46,968][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:38:47,465][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:38:47,956][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:38:48,450][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:38:48,943][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:38:49,435][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:38:49,927][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:38:50,419][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:38:50,911][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:38:51,404][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:38:51,896][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:38:52,392][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:38:52,885][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:38:53,376][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:38:53,867][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:38:54,362][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:38:54,858][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:38:55,362][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:38:55,855][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:38:56,358][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:38:56,850][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:38:57,342][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:38:57,841][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:38:58,334][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:38:58,842][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:38:59,335][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:38:59,826][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:39:00,326][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:39:00,816][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:39:01,308][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:39:01,800][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:39:02,293][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:39:02,785][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:39:03,283][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:39:03,776][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:39:04,270][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:39:04,764][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:39:05,263][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:39:05,754][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:39:06,247][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:39:06,756][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:39:07,250][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:39:07,746][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:39:08,241][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:39:08,733][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:39:09,244][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:39:09,735][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:39:10,230][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:39:10,728][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:39:11,222][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:39:11,722][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:39:12,217][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:39:12,709][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:39:13,211][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:39:13,705][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:39:14,200][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:39:14,695][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:39:15,189][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:39:15,683][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:39:16,176][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:39:16,670][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:39:17,170][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9885 tokens.
[2025-11-12 22:39:17,834][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.97%, Current % of VRAM taken: 60.23%, Block Peak % of device VRAM: 61.95%, ΔTime: 00:00:32
[2025-11-12 22:39:18,581][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:39:18,582][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:39:18,584][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:39:19,533][__main__][INFO] - Iteration 32 took 51s (30.95% Gen, 67.20% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 19m 26s. Estimated total time: 42h 49m 4s. Time estimates for 10 more iterations: 8m 33s, 100 more iterations: 1h 25m 38s, 500 more iterations: 7h 8m 10s.
[2025-11-12 22:39:19,536][__main__][INFO] - Starting iteration 32.
[2025-11-12 22:39:20,005][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:39:20,005][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:39:33,968][__main__][INFO] - Number of regex retries in iteration 32: 0
[2025-11-12 22:39:33,969][__main__][INFO] - agents played in iteration 32 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:39:34,781][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:39:34,809][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:39:34,835][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:39:34,858][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:39:34,861][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:39:34,862][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:39:35,481][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:39:35,933][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:39:36,455][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:39:36,982][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:39:37,478][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:39:37,978][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:39:38,469][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:39:38,968][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:39:39,460][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:39:39,952][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:39:40,460][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:39:40,954][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:39:41,449][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:39:41,942][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:39:42,439][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:39:42,955][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:39:43,451][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:39:43,946][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:39:44,441][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:39:44,935][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:39:45,434][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:39:45,927][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:39:46,419][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:39:46,913][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:39:47,408][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:39:47,902][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:39:48,398][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:39:48,893][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:39:49,391][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:39:49,887][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:39:50,382][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:39:50,902][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:39:51,405][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:39:51,900][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:39:52,394][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:39:52,887][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:39:53,385][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:39:53,881][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:39:54,376][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:39:54,870][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:39:55,368][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:39:55,866][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:39:56,367][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:39:56,866][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:39:57,371][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:39:57,865][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:39:58,364][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:39:58,860][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:39:59,357][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:39:59,861][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:40:00,356][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:40:00,851][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:40:01,367][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:40:01,865][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:40:02,360][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:40:02,855][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:40:03,348][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:40:03,855][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:40:04,348][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:40:04,841][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:40:05,338][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:40:05,830][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:40:06,324][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:40:06,817][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:40:07,310][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9936 tokens.
[2025-11-12 22:40:07,965][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 61.89%, ΔTime: 00:00:32
[2025-11-12 22:40:08,703][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:40:08,705][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:40:08,707][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:40:09,737][__main__][INFO] - Iteration 33 took 49s (28.08% Gen, 69.85% Train). Generation: 13s, Training: 34s. Estimated remaining time: 40h 56m 9s. Estimated total time: 41h 26m 37s. Time estimates for 10 more iterations: 8m 17s, 100 more iterations: 1h 22m 53s, 500 more iterations: 6h 54m 26s.
[2025-11-12 22:40:09,739][__main__][INFO] - Starting iteration 33.
[2025-11-12 22:40:10,216][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:40:10,217][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:40:11,594][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:40:24,638][__main__][INFO] - Number of regex retries in iteration 33: 1
[2025-11-12 22:40:24,638][__main__][INFO] - agents played in iteration 33 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:40:25,448][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:40:25,473][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:40:25,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:40:25,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:40:25,520][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:40:25,520][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:40:26,181][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:40:26,634][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:40:27,133][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:40:27,647][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:40:28,142][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:40:28,635][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:40:29,134][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:40:29,625][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:40:30,118][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:40:30,617][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:40:31,113][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:40:31,611][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:40:32,106][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:40:32,600][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:40:33,093][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:40:33,588][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:40:34,084][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:40:34,575][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:40:35,069][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:40:35,565][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:40:36,056][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:40:36,546][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:40:37,038][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:40:37,530][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:40:38,031][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:40:38,523][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:40:39,016][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:40:39,520][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:40:40,011][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:40:40,505][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:40:40,997][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:40:41,489][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:40:41,984][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:40:42,477][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:40:42,970][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:40:43,467][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:40:43,983][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:40:44,481][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:40:44,973][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:40:45,468][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:40:45,976][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:40:46,470][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:40:46,967][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:40:47,461][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:40:47,955][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:40:48,461][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:40:48,959][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:40:49,450][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:40:49,962][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:40:50,457][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:40:50,960][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:40:51,455][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:40:51,952][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:40:52,471][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:40:52,967][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:40:53,464][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:40:53,960][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:40:54,452][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:40:54,956][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:40:55,451][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:40:55,944][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:40:56,442][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:40:56,935][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:40:57,429][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:40:57,926][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9891 tokens.
[2025-11-12 22:40:58,576][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.01%, Current % of VRAM taken: 60.27%, Block Peak % of device VRAM: 61.88%, ΔTime: 00:00:32
[2025-11-12 22:40:59,334][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:40:59,336][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:40:59,338][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:41:00,417][__main__][INFO] - Iteration 34 took 50s (28.73% Gen, 69.12% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 18m 46s. Estimated total time: 41h 50m 5s. Time estimates for 10 more iterations: 8m 22s, 100 more iterations: 1h 23m 40s, 500 more iterations: 6h 58m 20s.
[2025-11-12 22:41:00,419][__main__][INFO] - Starting iteration 34.
[2025-11-12 22:41:00,914][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:41:00,915][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:41:14,747][__main__][INFO] - Number of regex retries in iteration 34: 0
[2025-11-12 22:41:14,747][__main__][INFO] - agents played in iteration 34 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:41:15,527][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:41:15,555][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:41:15,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:41:15,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:41:15,604][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:41:15,605][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:41:16,246][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:41:16,699][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:41:17,203][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:41:17,703][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:41:18,196][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:41:18,692][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:41:19,185][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:41:19,678][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:41:20,178][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:41:20,669][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:41:21,162][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:41:21,655][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:41:22,147][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:41:22,636][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:41:23,129][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:41:23,619][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:41:24,111][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:41:24,603][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:41:25,096][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:41:25,592][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:41:26,086][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:41:26,578][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:41:27,071][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:41:27,564][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:41:28,078][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:41:28,572][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:41:29,070][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:41:29,568][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:41:30,061][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:41:30,568][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:41:31,061][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:41:31,555][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:41:32,067][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:41:32,560][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:41:33,054][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:41:33,548][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:41:34,041][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:41:34,538][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:41:35,029][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:41:35,522][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:41:36,015][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:41:36,508][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:41:37,000][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:41:37,496][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:41:37,992][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:41:38,493][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:41:38,989][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:41:39,487][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:41:39,976][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:41:40,468][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:41:40,973][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:41:41,468][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:41:41,966][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:41:42,460][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:41:42,955][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:41:43,447][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:41:43,950][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:41:44,448][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:41:44,960][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:41:45,454][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:41:45,950][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:41:46,453][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:41:46,950][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:41:47,449][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:41:47,944][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9973 tokens.
[2025-11-12 22:41:48,623][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.09%, ΔTime: 00:00:32
[2025-11-12 22:41:49,385][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:41:49,386][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:41:49,388][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:41:50,357][__main__][INFO] - Iteration 35 took 49s (27.98% Gen, 70.06% Train). Generation: 13s, Training: 34s. Estimated remaining time: 40h 40m 1s. Estimated total time: 41h 12m 9s. Time estimates for 10 more iterations: 8m 14s, 100 more iterations: 1h 22m 24s, 500 more iterations: 6h 52m 1s.
[2025-11-12 22:41:50,359][__main__][INFO] - Starting iteration 35.
[2025-11-12 22:41:50,845][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:41:50,846][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:42:05,061][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 20 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:42:05,880][__main__][INFO] - Number of regex retries in iteration 35: 1
[2025-11-12 22:42:05,881][__main__][INFO] - agents played in iteration 35 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:42:06,744][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:42:06,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:42:06,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:42:06,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:42:06,819][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:42:06,820][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:42:07,466][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:42:07,920][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:42:08,415][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:42:08,906][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:42:09,398][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:42:09,892][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:42:10,387][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:42:10,878][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:42:11,369][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:42:11,865][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:42:12,359][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:42:12,853][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:42:13,348][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:42:13,888][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:42:14,380][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:42:14,876][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:42:15,372][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:42:15,866][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:42:16,358][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:42:16,852][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:42:17,347][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:42:17,853][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:42:18,347][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:42:18,840][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:42:19,346][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:42:19,838][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:42:20,328][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:42:20,820][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:42:21,315][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:42:21,808][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:42:22,300][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:42:22,789][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:42:23,283][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:42:23,773][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:42:24,264][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:42:24,753][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:42:25,245][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:42:25,749][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:42:26,240][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:42:26,729][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:42:27,221][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:42:27,713][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:42:28,204][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:42:28,701][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:42:29,195][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:42:29,696][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:42:30,193][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:42:30,687][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:42:31,188][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:42:31,687][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:42:32,183][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:42:32,694][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:42:33,189][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:42:33,685][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:42:34,177][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:42:34,674][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:42:35,181][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:42:35,676][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:42:36,171][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:42:36,672][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:42:37,169][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:42:37,663][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:42:38,159][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:42:38,656][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:42:39,158][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9959 tokens.
[2025-11-12 22:42:39,806][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 61.92%, ΔTime: 00:00:32
[2025-11-12 22:42:40,616][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:42:40,618][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:42:40,620][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:42:41,557][__main__][INFO] - Iteration 36 took 50s (29.65% Gen, 68.50% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 42m 38s. Estimated total time: 42h 15m 38s. Time estimates for 10 more iterations: 8m 27s, 100 more iterations: 1h 24m 31s, 500 more iterations: 7h 2m 36s.
[2025-11-12 22:42:41,579][__main__][INFO] - Starting iteration 36.
[2025-11-12 22:42:42,049][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:42:42,050][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:42:56,570][__main__][INFO] - Number of regex retries in iteration 36: 0
[2025-11-12 22:42:56,571][__main__][INFO] - agents played in iteration 36 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:42:57,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:42:57,422][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:42:57,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:42:57,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:42:57,478][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:42:57,479][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:42:58,114][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:42:58,570][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:42:59,075][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:42:59,581][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:43:00,081][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:43:00,577][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:43:01,073][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:43:01,585][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:43:02,078][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:43:02,575][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:43:03,074][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:43:03,571][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:43:04,084][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:43:04,578][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:43:05,073][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:43:05,571][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:43:06,067][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:43:06,559][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:43:07,056][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:43:07,551][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:43:08,057][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:43:08,549][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:43:09,039][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:43:09,530][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:43:10,024][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:43:10,519][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:43:11,013][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:43:11,507][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:43:12,006][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:43:12,499][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:43:13,002][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:43:13,494][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:43:13,990][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:43:14,497][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:43:14,990][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:43:15,486][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:43:15,987][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:43:16,483][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:43:16,989][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:43:17,484][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:43:17,979][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:43:18,475][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:43:18,969][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:43:19,463][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:43:19,961][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:43:20,460][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:43:20,962][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:43:21,457][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:43:21,951][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:43:22,460][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:43:22,956][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:43:23,452][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:43:23,947][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:43:24,442][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:43:24,936][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:43:25,430][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:43:25,926][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:43:26,425][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:43:26,919][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:43:27,414][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:43:27,907][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:43:28,403][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:43:28,895][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:43:29,392][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:43:29,885][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9870 tokens.
[2025-11-12 22:43:30,532][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.97%, Current % of VRAM taken: 60.23%, Block Peak % of device VRAM: 61.84%, ΔTime: 00:00:32
[2025-11-12 22:43:31,329][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:43:31,330][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:43:31,332][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:43:32,263][__main__][INFO] - Iteration 37 took 50s (28.92% Gen, 69.23% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 16m 51s. Estimated total time: 41h 50m 42s. Time estimates for 10 more iterations: 8m 22s, 100 more iterations: 1h 23m 41s, 500 more iterations: 6h 58m 27s.
[2025-11-12 22:43:32,265][__main__][INFO] - Starting iteration 37.
[2025-11-12 22:43:32,735][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:43:32,736][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:43:36,010][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:43:47,311][__main__][INFO] - Number of regex retries in iteration 37: 1
[2025-11-12 22:43:47,312][__main__][INFO] - agents played in iteration 37 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:43:48,152][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:43:48,181][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:43:48,207][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:43:48,231][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:43:48,231][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:43:48,232][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:43:48,834][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:43:49,285][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:43:49,788][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:43:50,284][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:43:50,786][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:43:51,284][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:43:51,779][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:43:52,288][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:43:52,782][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:43:53,274][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:43:53,771][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:43:54,263][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:43:54,753][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:43:55,245][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:43:55,735][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:43:56,230][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:43:56,724][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:43:57,217][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:43:57,723][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:43:58,234][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:43:58,734][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:43:59,229][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:43:59,723][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:44:00,223][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:44:00,719][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:44:01,212][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:44:01,704][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:44:02,195][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:44:02,695][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:44:03,187][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:44:03,680][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:44:04,190][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:44:04,683][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:44:05,176][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:44:05,673][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:44:06,166][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:44:06,679][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:44:07,170][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:44:07,662][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:44:08,154][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:44:08,645][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:44:09,134][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:44:09,626][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:44:10,121][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:44:10,619][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:44:11,111][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:44:11,607][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:44:12,098][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:44:12,590][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:44:13,083][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:44:13,577][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:44:14,074][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:44:14,593][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:44:15,088][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:44:15,583][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:44:16,081][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:44:16,574][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:44:17,078][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:44:17,577][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:44:18,070][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:44:18,579][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:44:19,078][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:44:19,571][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:44:20,071][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:44:20,567][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10000 tokens.
[2025-11-12 22:44:21,214][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.06%, ΔTime: 00:00:32
[2025-11-12 22:44:21,954][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:44:21,956][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:44:21,958][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:44:23,119][__main__][INFO] - Iteration 38 took 50s (28.93% Gen, 68.76% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 24m 33s. Estimated total time: 41h 59m 14s. Time estimates for 10 more iterations: 8m 23s, 100 more iterations: 1h 23m 58s, 500 more iterations: 6h 59m 52s.
[2025-11-12 22:44:23,122][__main__][INFO] - Starting iteration 38.
[2025-11-12 22:44:23,598][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:44:23,599][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:44:26,128][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:44:36,129][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 books, 10 balls, 10 hats did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:44:38,726][__main__][INFO] - Number of regex retries in iteration 38: 2
[2025-11-12 22:44:38,726][__main__][INFO] - agents played in iteration 38 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:44:39,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:44:39,592][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:44:39,618][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:44:39,642][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:44:39,642][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:44:39,644][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:44:40,296][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:44:40,754][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:44:41,257][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:44:41,753][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:44:42,249][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:44:42,743][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:44:43,239][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:44:43,735][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:44:44,228][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:44:44,728][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:44:45,219][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:44:45,711][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:44:46,204][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:44:46,696][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:44:47,189][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:44:47,683][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:44:48,175][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:44:48,694][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:44:49,191][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:44:49,684][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:44:50,182][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:44:50,675][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:44:51,170][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:44:51,663][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:44:52,154][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:44:52,650][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:44:53,144][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:44:53,635][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:44:54,128][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:44:54,619][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:44:55,114][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:44:55,612][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:44:56,106][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:44:56,609][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:44:57,107][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:44:57,601][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:44:58,094][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:44:58,590][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:44:59,097][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:44:59,591][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:45:00,090][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:45:00,588][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:45:01,090][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:45:01,586][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:45:02,084][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:45:02,577][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:45:03,098][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:45:03,593][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:45:04,087][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:45:04,585][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:45:05,081][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:45:05,580][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:45:06,074][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:45:06,571][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:45:07,069][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:45:07,570][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:45:08,067][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:45:08,572][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:45:09,065][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:45:09,560][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:45:10,053][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:45:10,548][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:45:11,043][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:45:11,538][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:45:12,031][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10045 tokens.
[2025-11-12 22:45:12,694][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.01%, Current % of VRAM taken: 60.27%, Block Peak % of device VRAM: 61.93%, ΔTime: 00:00:32
[2025-11-12 22:45:13,459][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:45:13,460][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:45:13,462][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:45:14,506][__main__][INFO] - Iteration 39 took 50s (29.72% Gen, 68.23% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 49m 49s. Estimated total time: 42h 25m 22s. Time estimates for 10 more iterations: 8m 29s, 100 more iterations: 1h 24m 50s, 500 more iterations: 7h 4m 13s.
[2025-11-12 22:45:14,508][__main__][INFO] - Starting iteration 39.
[2025-11-12 22:45:14,985][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:45:14,986][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:45:22,061][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given the item values:
- You value each hat at 10 points.
- Bob values each hat at 1 point.
- Both value books and balls equally at 1 point.

Since the value of hats is much higher for you compared to Bob, it makes the most sense to claim all the hats to maximize your points. Allocating any of the other items would result in a much lower value per item for you. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:45:29,952][__main__][INFO] - Number of regex retries in iteration 39: 1
[2025-11-12 22:45:29,953][__main__][INFO] - agents played in iteration 39 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:45:30,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:45:30,827][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:45:30,853][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:45:30,877][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:45:30,877][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:45:30,878][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:45:31,524][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:45:31,977][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:45:32,476][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:45:32,971][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:45:33,467][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:45:33,959][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:45:34,449][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:45:34,940][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:45:35,433][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:45:36,017][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:45:36,510][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:45:37,006][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:45:37,523][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:45:38,022][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:45:38,522][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:45:39,016][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:45:39,509][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:45:40,021][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:45:40,518][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:45:41,010][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:45:41,506][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:45:42,000][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:45:42,497][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:45:42,990][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:45:43,486][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:45:43,979][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:45:44,471][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:45:44,966][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:45:45,458][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:45:45,948][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:45:46,443][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:45:46,933][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:45:47,426][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:45:47,930][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:45:48,420][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:45:48,910][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:45:49,403][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:45:49,893][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:45:50,388][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:45:50,880][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:45:51,372][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:45:51,871][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:45:52,370][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:45:52,869][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:45:53,368][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:45:53,874][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:45:54,370][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:45:54,869][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:45:55,368][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:45:55,866][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:45:56,361][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:45:56,859][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:45:57,359][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:45:57,854][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:45:58,351][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:45:58,847][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:45:59,342][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:45:59,839][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:46:00,333][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:46:00,827][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:46:01,319][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:46:01,811][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:46:02,304][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:46:02,796][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:46:03,289][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9882 tokens.
[2025-11-12 22:46:03,936][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.01%, Current % of VRAM taken: 60.27%, Block Peak % of device VRAM: 62.00%, ΔTime: 00:00:32
[2025-11-12 22:46:04,693][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:46:04,694][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:46:04,696][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:46:05,644][__main__][INFO] - Iteration 40 took 50s (29.54% Gen, 68.58% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 36m 34s. Estimated total time: 42h 12m 58s. Time estimates for 10 more iterations: 8m 26s, 100 more iterations: 1h 24m 25s, 500 more iterations: 7h 2m 9s.
[2025-11-12 22:46:05,647][__main__][INFO] - Starting iteration 40.
[2025-11-12 22:46:06,138][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-12 22:46:06,138][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:46:07,483][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:46:07,486][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:46:20,260][__main__][INFO] - Number of regex retries in iteration 40: 2
[2025-11-12 22:46:20,261][__main__][INFO] - agents played in iteration 40 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:46:21,076][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:46:21,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:46:21,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:46:21,153][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:46:21,153][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:46:21,154][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:46:21,810][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:46:22,263][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:46:22,765][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:46:23,259][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:46:23,752][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:46:24,247][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:46:24,735][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:46:25,228][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:46:25,727][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:46:26,223][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:46:26,716][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:46:27,223][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:46:27,715][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:46:28,208][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:46:28,711][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:46:29,206][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:46:29,711][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:46:30,209][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:46:30,706][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:46:31,203][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:46:31,697][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:46:32,194][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:46:32,698][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:46:33,196][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:46:33,692][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:46:34,185][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:46:34,678][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:46:35,173][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:46:35,667][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:46:36,161][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:46:36,659][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:46:37,154][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:46:37,649][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:46:38,141][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:46:38,634][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:46:39,142][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:46:39,634][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:46:40,128][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:46:40,627][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:46:41,123][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:46:41,633][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:46:42,129][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:46:42,628][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:46:43,146][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:46:43,641][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:46:44,138][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:46:44,634][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:46:45,131][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:46:45,643][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:46:46,141][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:46:46,636][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:46:47,147][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:46:47,642][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:46:48,137][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:46:48,632][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:46:49,128][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:46:49,635][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:46:50,130][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:46:50,629][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:46:51,145][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:46:51,640][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:46:52,137][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:46:52,635][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:46:53,132][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:46:53,627][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9909 tokens.
[2025-11-12 22:46:54,266][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 61.78%, ΔTime: 00:00:32
[2025-11-12 22:46:55,044][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:46:55,046][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:46:55,048][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:46:57,510][__main__][INFO] - Iteration 41 took 51s (27.49% Gen, 67.71% Train). Generation: 14s, Training: 34s. Estimated remaining time: 42h 11m 24s. Estimated total time: 42h 48m 40s. Time estimates for 10 more iterations: 8m 33s, 100 more iterations: 1h 25m 37s, 500 more iterations: 7h 8m 6s.
[2025-11-12 22:46:57,512][__main__][INFO] - Starting iteration 41.
[2025-11-12 22:46:58,019][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:46:58,020][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:47:12,579][__main__][INFO] - Number of regex retries in iteration 41: 0
[2025-11-12 22:47:12,580][__main__][INFO] - agents played in iteration 41 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:47:13,396][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:47:13,421][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:47:13,447][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:47:13,470][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:47:13,470][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:47:13,471][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:47:14,130][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:47:14,587][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:47:15,088][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:47:15,585][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:47:16,081][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:47:16,575][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:47:17,070][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:47:17,566][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:47:18,060][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:47:18,554][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:47:19,049][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:47:19,542][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:47:20,047][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:47:20,545][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:47:21,038][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:47:21,534][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:47:22,028][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:47:22,522][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:47:23,024][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:47:23,520][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:47:24,012][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:47:24,506][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:47:25,000][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:47:25,496][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:47:25,990][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:47:26,484][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:47:26,978][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:47:27,472][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:47:27,970][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:47:28,465][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:47:28,960][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:47:29,461][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:47:29,964][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:47:30,461][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:47:30,953][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:47:31,446][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:47:31,953][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:47:32,451][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:47:32,947][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:47:33,447][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:47:33,946][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:47:34,453][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:47:34,948][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:47:35,442][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:47:35,952][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:47:36,448][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:47:36,946][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:47:37,442][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:47:37,939][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:47:38,436][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:47:38,933][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:47:39,432][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:47:39,931][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:47:40,428][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:47:40,924][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:47:41,432][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:47:41,927][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:47:42,438][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:47:42,933][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:47:43,455][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:47:43,950][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:47:44,448][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:47:44,956][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:47:45,460][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:47:45,956][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10029 tokens.
[2025-11-12 22:47:46,614][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:32
[2025-11-12 22:47:47,348][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:47:47,349][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:47:47,351][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:47:48,322][__main__][INFO] - Iteration 42 took 50s (28.94% Gen, 69.12% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 17m 2s. Estimated total time: 41h 55m 9s. Time estimates for 10 more iterations: 8m 23s, 100 more iterations: 1h 23m 50s, 500 more iterations: 6h 59m 11s.
[2025-11-12 22:47:48,324][__main__][INFO] - Starting iteration 42.
[2025-11-12 22:47:48,799][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:47:48,799][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:48:03,511][__main__][INFO] - Number of regex retries in iteration 42: 0
[2025-11-12 22:48:03,512][__main__][INFO] - agents played in iteration 42 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:48:04,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:48:04,408][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:48:04,434][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:48:04,458][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:48:04,458][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:48:04,459][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:48:05,109][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:48:05,561][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:48:06,065][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:48:06,557][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:48:07,051][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:48:07,549][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:48:08,046][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:48:08,539][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:48:09,030][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:48:09,524][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:48:10,023][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:48:10,518][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:48:11,010][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:48:11,508][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:48:12,003][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:48:12,495][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:48:12,989][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:48:13,483][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:48:13,978][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:48:14,471][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:48:14,967][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:48:15,457][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:48:15,948][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:48:16,453][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:48:16,945][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:48:17,438][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:48:17,948][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:48:18,443][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:48:18,937][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:48:19,440][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:48:19,934][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:48:20,429][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:48:20,924][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:48:21,418][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:48:21,911][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:48:22,404][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:48:22,898][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:48:23,395][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:48:23,891][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:48:24,386][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:48:24,882][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:48:25,377][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:48:25,879][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:48:26,375][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:48:26,870][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:48:27,369][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:48:27,865][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:48:28,368][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:48:28,863][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:48:29,358][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:48:29,873][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:48:30,370][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:48:30,868][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:48:31,367][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:48:31,861][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:48:32,370][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:48:32,865][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:48:33,361][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:48:33,857][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:48:34,353][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:48:34,859][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:48:35,356][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:48:35,850][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:48:36,349][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:48:36,845][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10071 tokens.
[2025-11-12 22:48:37,519][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 61.84%, ΔTime: 00:00:32
[2025-11-12 22:48:38,266][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:48:38,267][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:48:38,270][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:48:39,273][__main__][INFO] - Iteration 43 took 50s (29.15% Gen, 68.86% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 24m 47s. Estimated total time: 42h 3m 45s. Time estimates for 10 more iterations: 8m 24s, 100 more iterations: 1h 24m 7s, 500 more iterations: 7h 0m 37s.
[2025-11-12 22:48:39,275][__main__][INFO] - Starting iteration 43.
[2025-11-12 22:48:39,778][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:48:39,779][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:48:50,360][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given your per-item values for hats (10), books (1), and balls (1) compared to Bob's per-item values (hats: 1, books: 10, balls: 10), it makes the most sense to focus on claiming as many hats as possible since they are highly valuable to you. Books and balls are less valuable to you, and given Bob's higher valuation for books and balls, you should avoid proposing to take those items. This strategy aims to maximize the number of high-value items (hats) you receive. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:48:56,624][__main__][INFO] - Number of regex retries in iteration 43: 1
[2025-11-12 22:48:56,625][__main__][INFO] - agents played in iteration 43 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:48:57,403][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:48:57,426][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:48:57,448][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:48:57,470][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:48:57,470][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:48:57,471][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:48:58,059][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:48:58,509][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:48:59,010][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:48:59,505][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:49:00,003][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:49:00,495][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:49:00,987][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:49:01,490][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:49:01,983][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:49:02,481][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:49:02,975][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:49:03,471][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:49:03,972][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:49:04,484][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:49:04,981][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:49:05,486][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:49:05,982][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:49:06,481][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:49:06,975][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:49:07,475][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:49:07,978][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:49:08,476][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:49:08,969][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:49:09,474][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:49:09,969][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:49:10,465][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:49:10,958][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:49:11,476][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:49:11,991][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:49:12,485][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:49:12,980][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:49:13,481][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:49:13,977][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:49:14,470][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:49:14,968][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:49:15,468][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:49:15,985][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:49:16,481][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:49:16,979][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:49:17,476][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:49:17,971][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:49:18,476][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:49:18,971][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:49:19,464][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:49:19,980][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:49:20,475][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:49:20,970][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:49:21,467][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:49:21,962][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:49:22,477][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:49:22,972][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:49:23,467][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:49:23,964][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:49:24,458][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:49:24,969][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:49:25,466][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:49:25,960][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:49:26,474][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:49:26,970][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:49:27,464][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:49:27,959][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:49:28,455][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:49:28,968][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:49:29,465][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:49:29,962][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10056 tokens.
[2025-11-12 22:49:30,649][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:32
[2025-11-12 22:49:31,397][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:49:31,399][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:49:31,401][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:49:32,347][__main__][INFO] - Iteration 44 took 52s (32.05% Gen, 66.15% Train). Generation: 16s, Training: 34s. Estimated remaining time: 43h 8m 36s. Estimated total time: 43h 48m 26s. Time estimates for 10 more iterations: 8m 45s, 100 more iterations: 1h 27m 36s, 500 more iterations: 7h 18m 4s.
[2025-11-12 22:49:32,349][__main__][INFO] - Starting iteration 44.
[2025-11-12 22:49:32,845][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:49:32,846][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:49:48,303][__main__][INFO] - Number of regex retries in iteration 44: 0
[2025-11-12 22:49:48,304][__main__][INFO] - agents played in iteration 44 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:49:49,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:49:49,184][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:49:49,210][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:49:49,233][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:49:49,233][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:49:49,234][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:49:49,844][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:49:50,300][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:49:50,799][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:49:51,298][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:49:51,792][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:49:52,288][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:49:52,786][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:49:53,286][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:49:53,781][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:49:54,274][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:49:54,767][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:49:55,288][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:49:55,783][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:49:56,278][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:49:56,772][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:49:57,265][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:49:57,767][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:49:58,259][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:49:58,753][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:49:59,256][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:49:59,750][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:50:00,241][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:50:00,740][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:50:01,233][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:50:01,726][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:50:02,219][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:50:02,715][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:50:03,212][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:50:03,707][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:50:04,201][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:50:04,716][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:50:05,209][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:50:05,707][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:50:06,205][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:50:06,699][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:50:07,196][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:50:07,693][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:50:08,188][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:50:08,685][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:50:09,181][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:50:09,691][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:50:10,189][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:50:10,683][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:50:11,179][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:50:11,675][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:50:12,182][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:50:12,678][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:50:13,175][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:50:13,684][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:50:14,179][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:50:14,675][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:50:15,172][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:50:15,667][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:50:16,177][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:50:16,673][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:50:17,167][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:50:17,669][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:50:18,167][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:50:18,667][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:50:19,163][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:50:19,658][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:50:20,155][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:50:20,652][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:50:21,151][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:50:21,658][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10060 tokens.
[2025-11-12 22:50:22,346][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 61.98%, ΔTime: 00:00:32
[2025-11-12 22:50:23,142][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:50:23,158][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:50:23,160][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:50:24,136][__main__][INFO] - Iteration 45 took 51s (30.14% Gen, 67.96% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 3m 51s. Estimated total time: 42h 44m 34s. Time estimates for 10 more iterations: 8m 32s, 100 more iterations: 1h 25m 29s, 500 more iterations: 7h 7m 25s.
[2025-11-12 22:50:24,138][__main__][INFO] - Starting iteration 45.
[2025-11-12 22:50:24,607][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:50:24,607][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:50:39,515][__main__][INFO] - Number of regex retries in iteration 45: 0
[2025-11-12 22:50:39,516][__main__][INFO] - agents played in iteration 45 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:50:40,356][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:50:40,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:50:40,407][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:50:40,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:50:40,429][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:50:40,430][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:50:41,040][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:50:41,501][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:50:42,004][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:50:42,498][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:50:43,002][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:50:43,497][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:50:43,996][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:50:44,495][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:50:44,990][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:50:45,495][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:50:45,989][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:50:46,487][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:50:46,997][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:50:47,493][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:50:47,988][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:50:48,484][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:50:48,980][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:50:49,489][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:50:49,985][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:50:50,479][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:50:50,995][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:50:51,490][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:50:51,986][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:50:52,482][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:50:52,979][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:50:53,484][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:50:53,979][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:50:54,473][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:50:54,969][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:50:55,468][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:50:55,965][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:50:56,479][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:50:56,974][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:50:57,471][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:50:57,973][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:50:58,472][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:50:58,990][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:50:59,487][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:50:59,983][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:51:00,479][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:51:00,974][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:51:01,474][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:51:01,973][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:51:02,471][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:51:02,973][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:51:03,468][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:51:03,964][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:51:04,475][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:51:04,969][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:51:05,466][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:51:05,968][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:51:06,465][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:51:06,970][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:51:07,467][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:51:07,962][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:51:08,483][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:51:08,977][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:51:09,473][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:51:09,976][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:51:10,474][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:51:10,978][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:51:11,474][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:51:11,973][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:51:12,471][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:51:12,970][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10057 tokens.
[2025-11-12 22:51:13,661][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 61.94%, ΔTime: 00:00:32
[2025-11-12 22:51:14,423][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:51:14,425][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:51:14,426][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:51:15,374][__main__][INFO] - Iteration 46 took 50s (29.37% Gen, 68.77% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 36m 48s. Estimated total time: 42h 18m 22s. Time estimates for 10 more iterations: 8m 27s, 100 more iterations: 1h 24m 36s, 500 more iterations: 7h 3m 3s.
[2025-11-12 22:51:15,376][__main__][INFO] - Starting iteration 46.
[2025-11-12 22:51:15,853][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:51:15,853][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:51:17,378][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:51:30,314][__main__][INFO] - Number of regex retries in iteration 46: 1
[2025-11-12 22:51:30,314][__main__][INFO] - agents played in iteration 46 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:51:31,136][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:51:31,159][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:51:31,181][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:51:31,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:51:31,202][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:51:31,203][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:51:31,794][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:51:32,254][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:51:32,752][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:51:33,249][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:51:33,742][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:51:34,237][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:51:34,732][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:51:35,225][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:51:35,718][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:51:36,215][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:51:36,709][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:51:37,206][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:51:37,702][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:51:38,197][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:51:38,706][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:51:39,200][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:51:39,698][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:51:40,194][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:51:40,689][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:51:41,181][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:51:41,674][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:51:42,166][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:51:42,661][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:51:43,155][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:51:43,649][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:51:44,144][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:51:44,635][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:51:45,129][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:51:45,625][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:51:46,122][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:51:46,618][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:51:47,111][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:51:47,604][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:51:48,106][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:51:48,620][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:51:49,119][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:51:49,620][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:51:50,117][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:51:50,617][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:51:51,118][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:51:51,615][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:51:52,116][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:51:52,611][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:51:53,117][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:51:53,610][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:51:54,106][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:51:54,612][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:51:55,110][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:51:55,612][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:51:56,108][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:51:56,601][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:51:57,112][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:51:57,607][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:51:58,102][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:51:58,596][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:51:59,089][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:51:59,609][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:52:00,106][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:52:00,601][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:52:01,101][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:52:01,597][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:52:02,101][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:52:02,598][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:52:03,097][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:52:03,606][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10081 tokens.
[2025-11-12 22:52:04,300][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.02%, ΔTime: 00:00:32
[2025-11-12 22:52:05,076][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:52:05,078][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:52:05,079][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:52:06,057][__main__][INFO] - Iteration 47 took 50s (28.80% Gen, 69.25% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 7m 49s. Estimated total time: 41h 50m 13s. Time estimates for 10 more iterations: 8m 22s, 100 more iterations: 1h 23m 40s, 500 more iterations: 6h 58m 22s.
[2025-11-12 22:52:06,059][__main__][INFO] - Starting iteration 47.
[2025-11-12 22:52:06,531][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:52:06,532][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:52:10,019][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:52:10,259][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:52:21,238][__main__][INFO] - Number of regex retries in iteration 47: 2
[2025-11-12 22:52:21,239][__main__][INFO] - agents played in iteration 47 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:52:22,013][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:52:22,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:52:22,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:52:22,083][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:52:22,084][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:52:22,085][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:52:22,681][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:52:23,137][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:52:23,640][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:52:24,136][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:52:24,633][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:52:25,131][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:52:25,628][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:52:26,132][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:52:26,629][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:52:27,121][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:52:27,617][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:52:28,111][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:52:28,613][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:52:29,108][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:52:29,602][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:52:30,096][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:52:30,591][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:52:31,089][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:52:31,583][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:52:32,080][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:52:32,578][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:52:33,073][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:52:33,569][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:52:34,065][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:52:34,563][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:52:35,054][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:52:35,550][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:52:36,039][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:52:36,533][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:52:37,028][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:52:37,519][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:52:38,012][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:52:38,505][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:52:39,000][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:52:39,497][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:52:39,998][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:52:40,502][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:52:40,997][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:52:41,490][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:52:41,988][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:52:42,486][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:52:42,986][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:52:43,495][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:52:43,993][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:52:44,508][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:52:45,007][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:52:45,504][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:52:46,003][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:52:46,507][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:52:47,012][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:52:47,509][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:52:48,009][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:52:48,520][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:52:49,019][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:52:49,515][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:52:50,011][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:52:50,509][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:52:51,022][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:52:51,517][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:52:52,016][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:52:52,515][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:52:53,011][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:52:53,517][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:52:54,014][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:52:54,513][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10112 tokens.
[2025-11-12 22:52:55,202][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.02%, ΔTime: 00:00:32
[2025-11-12 22:52:55,978][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:52:55,980][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:52:55,982][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:52:56,940][__main__][INFO] - Iteration 48 took 50s (29.17% Gen, 68.92% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 17m 12s. Estimated total time: 42h 0m 27s. Time estimates for 10 more iterations: 8m 24s, 100 more iterations: 1h 24m 0s, 500 more iterations: 7h 0m 4s.
[2025-11-12 22:52:56,942][__main__][INFO] - Starting iteration 48.
[2025-11-12 22:52:57,419][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:52:57,419][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:53:05,150][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:53:11,992][__main__][INFO] - Number of regex retries in iteration 48: 1
[2025-11-12 22:53:11,992][__main__][INFO] - agents played in iteration 48 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:53:12,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:53:12,792][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:53:12,814][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:53:12,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:53:12,836][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:53:12,837][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:53:13,439][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:53:13,888][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:53:14,386][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:53:14,892][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:53:15,386][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:53:15,880][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:53:16,376][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:53:16,870][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:53:17,370][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:53:17,870][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:53:18,364][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:53:18,860][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:53:19,356][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:53:19,852][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:53:20,346][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:53:20,839][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:53:21,338][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:53:21,834][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:53:22,329][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:53:22,823][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:53:23,316][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:53:23,811][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:53:24,305][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:53:24,798][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:53:25,307][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:53:25,802][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:53:26,322][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:53:26,817][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:53:27,310][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:53:27,817][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:53:28,313][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:53:28,807][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:53:29,307][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:53:29,804][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:53:30,301][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:53:30,795][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:53:31,292][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:53:31,805][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:53:32,298][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:53:32,796][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:53:33,299][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:53:33,795][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:53:34,291][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:53:34,788][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:53:35,281][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:53:35,782][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:53:36,281][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:53:36,777][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:53:37,282][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:53:37,778][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:53:38,277][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:53:38,772][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:53:39,272][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:53:39,774][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:53:40,272][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:53:40,769][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:53:41,264][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:53:41,759][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:53:42,255][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:53:42,752][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:53:43,252][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:53:43,752][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:53:44,251][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:53:44,751][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:53:45,252][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10212 tokens.
[2025-11-12 22:53:45,961][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:32
[2025-11-12 22:53:46,742][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:53:46,744][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:53:46,746][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:53:47,725][__main__][INFO] - Iteration 49 took 50s (28.97% Gen, 69.08% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 11m 13s. Estimated total time: 41h 55m 19s. Time estimates for 10 more iterations: 8m 23s, 100 more iterations: 1h 23m 50s, 500 more iterations: 6h 59m 13s.
[2025-11-12 22:53:47,727][__main__][INFO] - Starting iteration 49.
[2025-11-12 22:53:48,215][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:53:48,215][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:53:49,899][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:54:03,507][__main__][INFO] - Number of regex retries in iteration 49: 1
[2025-11-12 22:54:03,508][__main__][INFO] - agents played in iteration 49 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:54:04,284][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:54:04,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:54:04,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:54:04,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:54:04,350][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:54:04,351][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:54:04,954][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:54:05,404][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:54:05,906][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:54:06,403][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:54:06,894][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:54:07,388][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:54:07,880][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:54:08,371][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:54:08,869][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:54:09,360][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:54:09,852][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:54:10,346][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:54:10,836][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:54:11,336][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:54:11,828][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:54:12,320][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:54:12,829][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:54:13,324][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:54:13,819][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:54:14,311][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:54:14,813][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:54:15,320][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:54:15,814][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:54:16,310][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:54:16,810][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:54:17,305][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:54:17,796][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:54:18,313][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:54:18,807][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:54:19,313][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:54:19,804][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:54:20,298][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:54:20,789][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:54:21,285][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:54:21,789][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:54:22,285][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:54:22,781][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:54:23,281][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:54:23,781][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:54:24,276][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:54:24,774][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:54:25,276][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:54:25,780][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:54:26,279][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:54:26,774][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:54:27,271][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:54:27,769][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:54:28,269][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:54:28,768][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:54:29,262][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:54:29,760][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:54:30,255][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:54:30,753][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:54:31,250][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:54:31,747][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:54:32,247][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:54:32,742][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:54:33,236][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:54:33,730][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:54:34,226][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:54:34,731][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:54:35,226][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:54:35,720][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:54:36,220][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:54:36,716][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10169 tokens.
[2025-11-12 22:54:37,400][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.06%, ΔTime: 00:00:32
[2025-11-12 22:54:38,192][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:54:38,193][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:54:38,195][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:54:39,167][__main__][INFO] - Iteration 50 took 50s (30.01% Gen, 68.08% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 42m 40s. Estimated total time: 42h 27m 38s. Time estimates for 10 more iterations: 8m 29s, 100 more iterations: 1h 24m 55s, 500 more iterations: 7h 4m 36s.
[2025-11-12 22:54:39,169][__main__][INFO] - Starting iteration 50.
[2025-11-12 22:54:39,657][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-12 22:54:39,658][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:54:54,814][__main__][INFO] - Number of regex retries in iteration 50: 0
[2025-11-12 22:54:54,814][__main__][INFO] - agents played in iteration 50 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:54:55,725][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:54:55,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:54:55,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:54:55,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:54:55,802][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:54:55,803][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:54:56,398][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:54:56,850][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:54:57,348][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:54:57,840][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:54:58,341][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:54:58,834][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:54:59,326][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:54:59,827][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:55:00,319][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:55:00,814][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:55:01,313][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:55:01,807][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:55:02,324][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:55:02,817][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:55:03,310][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:55:03,823][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:55:04,314][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:55:04,808][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:55:05,302][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:55:05,795][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:55:06,289][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:55:06,786][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:55:07,279][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:55:07,775][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:55:08,272][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:55:08,768][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:55:09,264][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:55:09,758][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:55:10,256][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:55:10,748][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:55:11,243][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:55:11,754][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:55:12,253][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:55:12,753][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:55:13,252][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:55:13,752][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:55:14,255][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:55:14,754][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:55:15,249][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:55:15,756][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:55:16,253][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:55:16,756][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:55:17,253][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:55:17,750][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:55:18,258][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:55:18,757][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:55:19,253][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:55:19,752][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:55:20,249][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:55:20,753][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:55:21,249][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:55:21,742][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:55:22,258][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:55:22,752][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:55:23,247][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:55:23,742][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:55:24,236][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:55:24,740][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:55:25,233][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:55:25,727][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:55:26,221][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:55:26,715][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:55:27,212][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:55:27,708][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:55:28,204][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10214 tokens.
[2025-11-12 22:55:28,903][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-12 22:55:29,695][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:55:29,697][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:55:29,698][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:55:31,712][__main__][INFO] - Iteration 51 took 52s (29.11% Gen, 67.01% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 36m 58s. Estimated total time: 43h 22m 48s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 45s, 500 more iterations: 7h 13m 48s.
[2025-11-12 22:55:31,715][__main__][INFO] - Starting iteration 51.
[2025-11-12 22:55:32,198][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 22:55:32,198][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:55:47,661][__main__][INFO] - Number of regex retries in iteration 51: 0
[2025-11-12 22:55:47,662][__main__][INFO] - agents played in iteration 51 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:55:48,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:55:48,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:55:48,587][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:55:48,610][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:55:48,610][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:55:48,611][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:55:49,207][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:55:49,666][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:55:50,166][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:55:50,663][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:55:51,159][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:55:51,653][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:55:52,146][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:55:52,643][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:55:53,139][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:55:53,649][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:55:54,146][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:55:54,652][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:55:55,151][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:55:55,647][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:55:56,142][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:55:56,638][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:55:57,134][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:55:57,639][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:55:58,133][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:55:58,628][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:55:59,151][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:55:59,646][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:56:00,140][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:56:00,642][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:56:01,135][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:56:01,631][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:56:02,127][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:56:02,625][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:56:03,121][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:56:03,618][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:56:04,115][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:56:04,615][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:56:05,115][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:56:05,612][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:56:06,112][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:56:06,610][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:56:07,110][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:56:07,608][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:56:08,106][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:56:08,608][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:56:09,103][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:56:09,602][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:56:10,098][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:56:10,592][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:56:11,091][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:56:11,589][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:56:12,086][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:56:12,582][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:56:13,078][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:56:13,579][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:56:14,074][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:56:14,571][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:56:15,069][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:56:15,563][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:56:16,070][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:56:16,565][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:56:17,059][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:56:17,573][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:56:18,071][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:56:18,574][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:56:19,070][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:56:19,568][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:56:20,079][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:56:20,580][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:56:21,082][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10192 tokens.
[2025-11-12 22:56:21,786][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-12 22:56:22,610][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:56:22,612][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:56:22,613][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:56:23,537][__main__][INFO] - Iteration 52 took 51s (30.12% Gen, 68.08% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 0m 18s. Estimated total time: 42h 47m 0s. Time estimates for 10 more iterations: 8m 33s, 100 more iterations: 1h 25m 34s, 500 more iterations: 7h 7m 50s.
[2025-11-12 22:56:23,539][__main__][INFO] - Starting iteration 52.
[2025-11-12 22:56:24,028][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 22:56:24,029][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:56:30,229][mllm.models.large_language_model_local][WARNING] - Response Given the new per-item values where hats and books are more valuable to you, a good strategy would be to maximize the items you value highly. Here’s the proposal:

Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:56:39,335][__main__][INFO] - Number of regex retries in iteration 52: 1
[2025-11-12 22:56:39,335][__main__][INFO] - agents played in iteration 52 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:56:40,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:56:40,196][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:56:40,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:56:40,246][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:56:40,246][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:56:40,247][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:56:40,854][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:56:41,306][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:56:41,806][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:56:42,302][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:56:42,803][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:56:43,297][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:56:43,793][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:56:44,288][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:56:44,784][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:56:45,278][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:56:45,771][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:56:46,274][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:56:46,768][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:56:47,262][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:56:47,756][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:56:48,247][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:56:48,741][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:56:49,236][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:56:49,731][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:56:50,230][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:56:50,725][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:56:51,218][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:56:51,714][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:56:52,208][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:56:52,703][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:56:53,198][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:56:53,689][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:56:54,184][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:56:54,680][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:56:55,176][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:56:55,679][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:56:56,175][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:56:56,671][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:56:57,173][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:56:57,669][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:56:58,171][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:56:58,667][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:56:59,164][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:56:59,676][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:57:00,171][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:57:00,667][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:57:01,165][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:57:01,661][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:57:02,168][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:57:02,664][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:57:03,159][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:57:03,677][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:57:04,173][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:57:04,669][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:57:05,172][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:57:05,667][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:57:06,175][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:57:06,672][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:57:07,168][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:57:07,665][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:57:08,164][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:57:08,670][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:57:09,169][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:57:09,665][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:57:10,179][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:57:10,677][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:57:11,174][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:57:11,674][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:57:12,176][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:57:12,682][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10192 tokens.
[2025-11-12 22:57:13,387][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 61.87%, ΔTime: 00:00:32
[2025-11-12 22:57:14,179][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:57:14,182][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:57:14,183][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:57:15,344][__main__][INFO] - Iteration 53 took 51s (29.83% Gen, 67.91% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 58m 14s. Estimated total time: 42h 45m 48s. Time estimates for 10 more iterations: 8m 33s, 100 more iterations: 1h 25m 31s, 500 more iterations: 7h 7m 38s.
[2025-11-12 22:57:15,346][__main__][INFO] - Starting iteration 53.
[2025-11-12 22:57:15,819][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 22:57:15,820][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:57:28,532][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:57:30,816][__main__][INFO] - Number of regex retries in iteration 53: 1
[2025-11-12 22:57:30,816][__main__][INFO] - agents played in iteration 53 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:57:31,595][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:57:31,621][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:57:31,647][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:57:31,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:57:31,671][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:57:31,671][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:57:32,259][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:57:32,717][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:57:33,212][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:57:33,710][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:57:34,203][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:57:34,696][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:57:35,192][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:57:35,686][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:57:36,181][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:57:36,685][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:57:37,179][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:57:37,673][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:57:38,173][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:57:38,669][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:57:39,172][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:57:39,669][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:57:40,162][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:57:40,678][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:57:41,171][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:57:41,664][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:57:42,160][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:57:42,656][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:57:43,176][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:57:43,670][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:57:44,163][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:57:44,678][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:57:45,173][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:57:45,671][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:57:46,171][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:57:46,670][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:57:47,186][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:57:47,680][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:57:48,177][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:57:48,673][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:57:49,169][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:57:49,671][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:57:50,169][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:57:50,668][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:57:51,169][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:57:51,667][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:57:52,163][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:57:52,662][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:57:53,165][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:57:53,687][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:57:54,189][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:57:54,687][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:57:55,187][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:57:55,684][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:57:56,182][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:57:56,675][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:57:57,169][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:57:57,678][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:57:58,171][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:57:58,666][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:57:59,167][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:57:59,663][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:58:00,160][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:58:00,657][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:58:01,158][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:58:01,657][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:58:02,156][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:58:02,657][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:58:03,169][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:58:03,669][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:58:04,184][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10144 tokens.
[2025-11-12 22:58:04,868][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:32
[2025-11-12 22:58:05,662][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:58:05,664][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:58:05,665][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:58:06,636][__main__][INFO] - Iteration 54 took 50s (29.51% Gen, 68.58% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 32m 28s. Estimated total time: 42h 20m 53s. Time estimates for 10 more iterations: 8m 28s, 100 more iterations: 1h 24m 41s, 500 more iterations: 7h 3m 28s.
[2025-11-12 22:58:06,639][__main__][INFO] - Starting iteration 54.
[2025-11-12 22:58:07,110][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 22:58:07,110][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:58:09,565][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:58:21,682][__main__][INFO] - Number of regex retries in iteration 54: 1
[2025-11-12 22:58:21,682][__main__][INFO] - agents played in iteration 54 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:58:22,485][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:58:22,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:58:22,533][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:58:22,555][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:58:22,556][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:58:22,557][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:58:23,145][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:58:23,600][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:58:24,107][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:58:24,601][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:58:25,102][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:58:25,598][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:58:26,093][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:58:26,590][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:58:27,090][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:58:27,584][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:58:28,079][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:58:28,573][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:58:29,080][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:58:29,574][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:58:30,070][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:58:30,577][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:58:31,071][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:58:31,565][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:58:32,057][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:58:32,551][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:58:33,046][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:58:33,539][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:58:34,031][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:58:34,528][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:58:35,021][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:58:35,514][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:58:36,012][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:58:36,509][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:58:37,007][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:58:37,502][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:58:37,998][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:58:38,494][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:58:38,993][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:58:39,492][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:58:39,990][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:58:40,486][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:58:40,997][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:58:41,495][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:58:41,994][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:58:42,493][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:58:42,988][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:58:43,491][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:58:43,988][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:58:44,484][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:58:44,981][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:58:45,476][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:58:45,973][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:58:46,467][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:58:46,962][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:58:47,469][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:58:47,966][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:58:48,460][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:58:48,968][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:58:49,462][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:58:49,957][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:58:50,450][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:58:50,948][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:58:51,442][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:58:51,936][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:58:52,429][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:58:52,922][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:58:53,416][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:58:53,917][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:58:54,412][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:58:54,910][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10130 tokens.
[2025-11-12 22:58:55,586][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-12 22:58:56,387][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:58:56,388][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:58:56,390][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:58:57,495][__main__][INFO] - Iteration 55 took 50s (28.92% Gen, 68.88% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 10m 0s. Estimated total time: 41h 59m 16s. Time estimates for 10 more iterations: 8m 23s, 100 more iterations: 1h 23m 58s, 500 more iterations: 6h 59m 52s.
[2025-11-12 22:58:57,497][__main__][INFO] - Starting iteration 55.
[2025-11-12 22:58:58,006][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 22:58:58,006][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:58:59,957][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:59:02,875][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:59:03,220][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 22:59:13,797][__main__][INFO] - Number of regex retries in iteration 55: 3
[2025-11-12 22:59:13,798][__main__][INFO] - agents played in iteration 55 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 22:59:14,569][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:59:14,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:59:14,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:59:14,639][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 22:59:14,640][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 22:59:14,641][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 22:59:15,257][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 22:59:15,709][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 22:59:16,207][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 22:59:16,702][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 22:59:17,195][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 22:59:17,704][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 22:59:18,199][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 22:59:18,691][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 22:59:19,186][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 22:59:19,689][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 22:59:20,194][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 22:59:20,694][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 22:59:21,189][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 22:59:21,687][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 22:59:22,189][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 22:59:22,685][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 22:59:23,185][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 22:59:23,678][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 22:59:24,195][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 22:59:24,691][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 22:59:25,185][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 22:59:25,697][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 22:59:26,191][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 22:59:26,687][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 22:59:27,184][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 22:59:27,682][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 22:59:28,199][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 22:59:28,697][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 22:59:29,194][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 22:59:29,693][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 22:59:30,191][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 22:59:30,698][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 22:59:31,194][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 22:59:31,690][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 22:59:32,206][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 22:59:32,703][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 22:59:33,201][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 22:59:33,699][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 22:59:34,197][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 22:59:34,706][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 22:59:35,203][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 22:59:35,708][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 22:59:36,208][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 22:59:36,705][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 22:59:37,234][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 22:59:37,732][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 22:59:38,231][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 22:59:38,732][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 22:59:39,234][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 22:59:39,736][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 22:59:40,232][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 22:59:40,731][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 22:59:41,230][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 22:59:41,731][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 22:59:42,233][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 22:59:42,733][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 22:59:43,233][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 22:59:43,734][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 22:59:44,233][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 22:59:44,732][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 22:59:45,233][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 22:59:45,738][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 22:59:46,237][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 22:59:46,738][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 22:59:47,238][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10259 tokens.
[2025-11-12 22:59:47,926][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-12 22:59:48,676][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 22:59:48,677][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 22:59:48,679][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 22:59:49,621][__main__][INFO] - Iteration 56 took 51s (30.59% Gen, 67.58% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 10m 40s. Estimated total time: 43h 0m 48s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 1s, 500 more iterations: 7h 10m 8s.
[2025-11-12 22:59:49,624][__main__][INFO] - Starting iteration 56.
[2025-11-12 22:59:50,099][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 22:59:50,099][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 22:59:57,652][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:00:05,889][__main__][INFO] - Number of regex retries in iteration 56: 1
[2025-11-12 23:00:05,889][__main__][INFO] - agents played in iteration 56 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:00:06,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:00:06,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:00:06,764][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:00:06,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:00:06,786][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:00:06,787][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:00:07,395][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:00:07,850][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:00:08,351][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:00:08,851][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:00:09,349][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:00:09,850][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:00:10,350][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:00:10,847][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:00:11,347][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:00:11,842][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:00:12,343][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:00:12,841][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:00:13,338][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:00:13,832][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:00:14,330][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:00:14,825][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:00:15,327][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:00:15,820][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:00:16,313][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:00:16,807][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:00:17,301][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:00:17,798][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:00:18,291][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:00:18,788][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:00:19,299][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:00:19,794][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:00:20,294][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:00:20,796][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:00:21,296][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:00:21,803][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:00:22,298][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:00:22,798][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:00:23,300][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:00:23,794][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:00:24,290][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:00:24,788][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:00:25,284][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:00:25,780][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:00:26,276][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:00:26,774][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:00:27,271][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:00:27,769][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:00:28,263][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:00:28,758][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:00:29,254][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:00:29,751][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:00:30,250][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:00:30,745][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:00:31,248][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:00:31,745][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:00:32,245][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:00:32,739][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:00:33,235][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:00:33,734][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:00:34,232][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:00:34,731][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:00:35,231][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:00:35,729][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:00:36,226][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:00:36,719][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:00:37,218][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:00:37,718][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:00:38,213][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:00:38,715][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:00:39,216][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10202 tokens.
[2025-11-12 23:00:39,882][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:32
[2025-11-12 23:00:40,645][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:00:40,647][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:00:40,648][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:00:41,613][__main__][INFO] - Iteration 57 took 51s (30.65% Gen, 67.47% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 4m 44s. Estimated total time: 42h 55m 44s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 51s, 500 more iterations: 7h 9m 17s.
[2025-11-12 23:00:41,615][__main__][INFO] - Starting iteration 57.
[2025-11-12 23:00:42,101][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 23:00:42,102][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:00:57,565][__main__][INFO] - Number of regex retries in iteration 57: 0
[2025-11-12 23:00:57,566][__main__][INFO] - agents played in iteration 57 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:00:58,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:00:58,441][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:00:58,469][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:00:58,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:00:58,496][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:00:58,496][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:00:59,101][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:00:59,550][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:01:00,055][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:01:00,550][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:01:01,047][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:01:01,543][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:01:02,037][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:01:02,532][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:01:03,029][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:01:03,522][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:01:04,023][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:01:04,516][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:01:05,011][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:01:05,523][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:01:06,016][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:01:06,511][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:01:07,007][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:01:07,505][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:01:08,011][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:01:08,505][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:01:08,999][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:01:09,513][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:01:10,006][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:01:10,503][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:01:11,001][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:01:11,496][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:01:12,013][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:01:12,508][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:01:13,001][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:01:13,513][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:01:14,010][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:01:14,506][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:01:15,002][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:01:15,498][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:01:16,008][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:01:16,504][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:01:17,001][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:01:17,521][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:01:18,019][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:01:18,515][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:01:19,011][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:01:19,514][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:01:20,033][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:01:20,531][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:01:21,030][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:01:21,555][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:01:22,058][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:01:22,555][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:01:23,058][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:01:23,555][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:01:24,054][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:01:24,549][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:01:25,047][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:01:25,549][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:01:26,047][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:01:26,545][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:01:27,039][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:01:27,533][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:01:28,035][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:01:28,530][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:01:29,023][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:01:29,525][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:01:30,021][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:01:30,526][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:01:31,022][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10162 tokens.
[2025-11-12 23:01:31,725][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-12 23:01:32,654][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:01:32,656][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:01:32,672][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:01:33,828][__main__][INFO] - Iteration 58 took 51s (29.90% Gen, 67.87% Train). Generation: 15s, Training: 35s. Estimated remaining time: 42h 14m 32s. Estimated total time: 43h 6m 24s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 12s, 500 more iterations: 7h 11m 4s.
[2025-11-12 23:01:33,830][__main__][INFO] - Starting iteration 58.
[2025-11-12 23:01:34,311][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 23:01:34,311][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:01:36,186][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:01:49,088][__main__][INFO] - Number of regex retries in iteration 58: 1
[2025-11-12 23:01:49,089][__main__][INFO] - agents played in iteration 58 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:01:49,919][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:01:49,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:01:49,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:01:50,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:01:50,003][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:01:50,004][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:01:50,610][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:01:51,067][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:01:51,569][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:01:52,065][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:01:52,560][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:01:53,057][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:01:53,553][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:01:54,049][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:01:54,549][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:01:55,045][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:01:55,540][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:01:56,034][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:01:56,529][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:01:57,023][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:01:57,517][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:01:58,030][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:01:58,527][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:01:59,024][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:01:59,521][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:02:00,019][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:02:00,522][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:02:01,019][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:02:01,515][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:02:02,026][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:02:02,522][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:02:03,017][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:02:03,516][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:02:04,012][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:02:04,522][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:02:05,016][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:02:05,515][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:02:06,021][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:02:06,518][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:02:07,030][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:02:07,526][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:02:08,021][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:02:08,535][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:02:09,032][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:02:09,530][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:02:10,030][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:02:10,527][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:02:11,035][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:02:11,532][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:02:12,028][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:02:12,530][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:02:13,028][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:02:13,535][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:02:14,033][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:02:14,531][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:02:15,029][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:02:15,530][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:02:16,029][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:02:16,523][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:02:17,019][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:02:17,530][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:02:18,024][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:02:18,522][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:02:19,019][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:02:19,515][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:02:20,028][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:02:20,523][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:02:21,021][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:02:21,534][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:02:22,031][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:02:22,529][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10254 tokens.
[2025-11-12 23:02:23,223][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 61.94%, ΔTime: 00:00:32
[2025-11-12 23:02:23,966][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:02:23,968][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:02:23,969][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:02:24,935][__main__][INFO] - Iteration 59 took 50s (29.19% Gen, 68.90% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 18m 31s. Estimated total time: 42h 11m 14s. Time estimates for 10 more iterations: 8m 26s, 100 more iterations: 1h 24m 22s, 500 more iterations: 7h 1m 52s.
[2025-11-12 23:02:24,937][__main__][INFO] - Starting iteration 59.
[2025-11-12 23:02:25,405][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 23:02:25,406][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:02:40,507][__main__][INFO] - Number of regex retries in iteration 59: 0
[2025-11-12 23:02:40,508][__main__][INFO] - agents played in iteration 59 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:02:41,361][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:02:41,388][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:02:41,414][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:02:41,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:02:41,437][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:02:41,438][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:02:42,038][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:02:42,488][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:02:42,992][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:02:43,492][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:02:43,992][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:02:44,488][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:02:44,986][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:02:45,482][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:02:45,982][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:02:46,477][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:02:46,974][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:02:47,469][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:02:47,971][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:02:48,468][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:02:48,966][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:02:49,469][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:02:49,966][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:02:50,467][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:02:50,961][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:02:51,456][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:02:51,951][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:02:52,444][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:02:52,940][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:02:53,434][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:02:53,927][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:02:54,420][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:02:54,918][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:02:55,414][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:02:55,935][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:02:56,431][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:02:56,930][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:02:57,450][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:02:57,946][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:02:58,443][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:02:58,942][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:02:59,440][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:02:59,945][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:03:00,442][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:03:00,936][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:03:01,444][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:03:01,938][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:03:02,439][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:03:02,932][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:03:03,425][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:03:03,922][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:03:04,422][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:03:04,921][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:03:05,438][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:03:05,936][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:03:06,431][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:03:06,927][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:03:07,422][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:03:07,940][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:03:08,438][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:03:08,936][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:03:09,443][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:03:09,937][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:03:10,431][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:03:10,926][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:03:11,421][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:03:11,927][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:03:12,423][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:03:12,925][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:03:13,440][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:03:13,938][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10207 tokens.
[2025-11-12 23:03:14,652][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.01%, ΔTime: 00:00:32
[2025-11-12 23:03:15,393][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:03:15,395][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:03:15,397][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:03:16,368][__main__][INFO] - Iteration 60 took 50s (29.63% Gen, 68.46% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 34m 36s. Estimated total time: 42h 28m 10s. Time estimates for 10 more iterations: 8m 29s, 100 more iterations: 1h 24m 56s, 500 more iterations: 7h 4m 41s.
[2025-11-12 23:03:16,370][__main__][INFO] - Starting iteration 60.
[2025-11-12 23:03:16,861][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-12 23:03:16,862][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:03:32,328][__main__][INFO] - Number of regex retries in iteration 60: 0
[2025-11-12 23:03:32,329][__main__][INFO] - agents played in iteration 60 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:03:33,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:03:33,148][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:03:33,173][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:03:33,195][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:03:33,196][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:03:33,197][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:03:33,793][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:03:34,245][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:03:34,743][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:03:35,237][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:03:35,862][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:03:36,362][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:03:36,857][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:03:37,352][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:03:37,848][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:03:38,355][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:03:38,858][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:03:39,356][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:03:39,858][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:03:40,357][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:03:40,860][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:03:41,357][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:03:41,855][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:03:42,353][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:03:42,850][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:03:43,349][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:03:43,843][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:03:44,336][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:03:44,849][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:03:45,346][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:03:45,843][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:03:46,340][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:03:46,833][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:03:47,327][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:03:47,829][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:03:48,322][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:03:48,826][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:03:49,325][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:03:49,825][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:03:50,325][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:03:50,828][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:03:51,327][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:03:51,828][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:03:52,325][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:03:52,843][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:03:53,339][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:03:53,834][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:03:54,331][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:03:54,828][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:03:55,336][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:03:55,831][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:03:56,325][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:03:56,825][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:03:57,321][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:03:57,818][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:03:58,326][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:03:58,822][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:03:59,323][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:03:59,816][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:04:00,308][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:04:00,803][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:04:01,297][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:04:01,790][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:04:02,310][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:04:02,806][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:04:03,305][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:04:03,801][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:04:04,297][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:04:04,798][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:04:05,297][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:04:05,796][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10217 tokens.
[2025-11-12 23:04:06,456][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-12 23:04:07,236][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:04:07,238][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:04:07,241][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:04:09,060][__main__][INFO] - Iteration 61 took 52s (29.63% Gen, 66.88% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 35m 30s. Estimated total time: 43h 29m 58s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 59s, 500 more iterations: 7h 14m 59s.
[2025-11-12 23:04:09,063][__main__][INFO] - Starting iteration 61.
[2025-11-12 23:04:09,553][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:04:09,554][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:04:12,028][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:04:25,668][__main__][INFO] - Number of regex retries in iteration 61: 1
[2025-11-12 23:04:25,669][__main__][INFO] - agents played in iteration 61 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:04:26,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:04:26,481][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:04:26,506][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:04:26,529][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:04:26,530][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:04:26,530][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:04:27,139][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:04:27,590][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:04:28,090][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:04:28,588][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:04:29,083][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:04:29,577][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:04:30,072][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:04:30,567][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:04:31,067][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:04:31,568][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:04:32,071][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:04:32,568][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:04:33,074][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:04:33,578][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:04:34,074][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:04:34,573][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:04:35,072][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:04:35,566][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:04:36,059][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:04:36,557][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:04:37,052][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:04:37,562][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:04:38,057][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:04:38,552][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:04:39,049][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:04:39,542][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:04:40,045][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:04:40,539][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:04:41,034][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:04:41,546][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:04:42,040][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:04:42,536][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:04:43,037][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:04:43,533][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:04:44,036][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:04:44,535][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:04:45,032][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:04:45,532][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:04:46,031][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:04:46,527][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:04:47,032][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:04:47,530][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:04:48,027][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:04:48,528][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:04:49,025][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:04:49,520][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:04:50,014][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:04:50,513][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:04:51,009][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:04:51,505][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:04:52,001][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:04:52,497][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:04:52,990][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:04:53,488][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:04:53,981][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:04:54,476][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:04:54,973][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:04:55,472][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:04:55,973][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:04:56,469][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:04:56,966][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:04:57,463][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:04:57,955][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:04:58,453][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:04:58,953][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10275 tokens.
[2025-11-12 23:04:59,657][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.02%, ΔTime: 00:00:32
[2025-11-12 23:05:00,423][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:05:00,425][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:05:00,427][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:05:01,499][__main__][INFO] - Iteration 62 took 51s (31.02% Gen, 66.91% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 21m 57s. Estimated total time: 43h 17m 17s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 34s, 500 more iterations: 7h 12m 52s.
[2025-11-12 23:05:01,501][__main__][INFO] - Starting iteration 62.
[2025-11-12 23:05:02,006][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:05:02,006][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:05:17,725][__main__][INFO] - Number of regex retries in iteration 62: 0
[2025-11-12 23:05:17,726][__main__][INFO] - agents played in iteration 62 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:05:18,516][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:05:18,543][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:05:18,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:05:18,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:05:18,593][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:05:18,594][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:05:19,192][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:05:19,659][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:05:20,155][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:05:20,653][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:05:21,153][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:05:21,656][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:05:22,164][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:05:22,661][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:05:23,158][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:05:23,673][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:05:24,170][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:05:24,666][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:05:25,161][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:05:25,655][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:05:26,149][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:05:26,647][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:05:27,144][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:05:27,648][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:05:28,147][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:05:28,641][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:05:29,155][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:05:29,657][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:05:30,161][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:05:30,657][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:05:31,156][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:05:31,652][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:05:32,146][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:05:32,645][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:05:33,140][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:05:33,635][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:05:34,143][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:05:34,638][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:05:35,135][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:05:35,641][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:05:36,139][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:05:36,638][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:05:37,137][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:05:37,637][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:05:38,137][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:05:38,638][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:05:39,139][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:05:39,636][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:05:40,133][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:05:40,653][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:05:41,148][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:05:41,644][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:05:42,141][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:05:42,636][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:05:43,131][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:05:43,633][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:05:44,132][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:05:44,657][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:05:45,156][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:05:45,652][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:05:46,154][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:05:46,647][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:05:47,147][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:05:47,641][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:05:48,136][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:05:48,634][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:05:49,133][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:05:49,628][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:05:50,130][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:05:50,628][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:05:51,127][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10149 tokens.
[2025-11-12 23:05:51,806][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:32
[2025-11-12 23:05:52,607][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:05:52,609][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:05:52,611][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:05:53,771][__main__][INFO] - Iteration 63 took 51s (30.37% Gen, 67.39% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 12m 6s. Estimated total time: 43h 8m 18s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 16s, 500 more iterations: 7h 11m 23s.
[2025-11-12 23:05:53,774][__main__][INFO] - Starting iteration 63.
[2025-11-12 23:05:54,258][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:05:54,258][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:06:09,787][__main__][INFO] - Number of regex retries in iteration 63: 0
[2025-11-12 23:06:09,787][__main__][INFO] - agents played in iteration 63 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:06:10,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:06:10,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:06:10,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:06:10,760][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:06:10,761][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:06:10,762][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:06:11,429][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:06:11,885][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:06:12,391][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:06:12,890][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:06:13,399][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:06:13,896][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:06:14,402][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:06:14,900][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:06:15,396][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:06:15,894][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:06:16,391][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:06:16,884][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:06:17,388][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:06:17,884][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:06:18,382][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:06:18,877][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:06:19,372][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:06:19,875][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:06:20,373][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:06:20,870][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:06:21,368][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:06:21,863][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:06:22,361][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:06:22,858][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:06:23,353][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:06:23,851][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:06:24,350][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:06:24,849][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:06:25,359][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:06:25,856][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:06:26,353][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:06:26,856][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:06:27,354][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:06:27,855][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:06:28,350][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:06:28,845][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:06:29,351][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:06:29,848][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:06:30,348][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:06:30,844][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:06:31,340][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:06:31,835][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:06:32,330][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:06:32,824][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:06:33,320][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:06:33,814][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:06:34,313][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:06:34,808][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:06:35,305][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:06:35,800][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:06:36,294][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:06:36,790][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:06:37,285][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:06:37,778][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:06:38,273][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:06:38,766][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:06:39,259][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:06:39,767][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:06:40,261][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:06:40,753][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:06:41,263][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:06:41,756][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:06:42,251][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:06:42,747][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:06:43,243][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10136 tokens.
[2025-11-12 23:06:43,924][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:32
[2025-11-12 23:06:44,688][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:06:44,690][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:06:44,692][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:06:45,647][__main__][INFO] - Iteration 64 took 51s (30.22% Gen, 67.92% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 52m 25s. Estimated total time: 42h 49m 29s. Time estimates for 10 more iterations: 8m 33s, 100 more iterations: 1h 25m 38s, 500 more iterations: 7h 8m 14s.
[2025-11-12 23:06:45,649][__main__][INFO] - Starting iteration 64.
[2025-11-12 23:06:46,210][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:06:46,211][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:06:48,327][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 5 hats, 5 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:07:01,421][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 1 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:07:02,205][__main__][INFO] - Number of regex retries in iteration 64: 2
[2025-11-12 23:07:02,205][__main__][INFO] - agents played in iteration 64 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:07:03,166][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:07:03,194][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:07:03,230][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:07:03,256][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:07:03,257][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:07:03,258][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:07:03,957][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:07:04,411][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:07:04,922][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:07:05,417][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:07:05,915][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:07:06,419][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:07:06,917][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:07:07,414][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:07:07,912][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:07:08,413][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:07:08,916][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:07:09,411][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:07:09,907][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:07:10,406][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:07:10,901][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:07:11,408][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:07:11,903][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:07:12,398][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:07:12,908][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:07:13,404][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:07:13,899][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:07:14,395][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:07:14,893][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:07:15,402][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:07:15,902][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:07:16,401][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:07:16,909][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:07:17,405][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:07:17,903][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:07:18,400][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:07:18,896][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:07:19,398][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:07:19,894][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:07:20,388][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:07:20,905][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:07:21,403][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:07:21,902][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:07:22,397][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:07:22,891][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:07:23,402][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:07:23,899][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:07:24,391][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:07:24,900][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:07:25,392][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:07:25,884][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:07:26,382][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:07:26,874][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:07:27,371][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:07:27,864][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:07:28,359][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:07:28,860][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:07:29,357][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:07:29,850][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:07:30,349][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:07:30,842][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:07:31,339][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:07:31,838][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:07:32,337][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:07:32,836][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:07:33,335][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:07:33,835][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:07:34,340][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:07:34,841][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:07:35,335][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:07:35,831][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10186 tokens.
[2025-11-12 23:07:36,490][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.04%, ΔTime: 00:00:32
[2025-11-12 23:07:37,276][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:07:37,278][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:07:37,279][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:07:38,262][__main__][INFO] - Iteration 65 took 52s (30.73% Gen, 67.38% Train). Generation: 15s, Training: 35s. Estimated remaining time: 42h 24m 39s. Estimated total time: 43h 22m 35s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 45s, 500 more iterations: 7h 13m 45s.
[2025-11-12 23:07:38,264][__main__][INFO] - Starting iteration 65.
[2025-11-12 23:07:38,768][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:07:38,769][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:07:54,351][__main__][INFO] - Number of regex retries in iteration 65: 0
[2025-11-12 23:07:54,352][__main__][INFO] - agents played in iteration 65 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:07:55,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:07:55,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:07:55,353][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:07:55,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:07:55,376][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:07:55,377][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:07:56,037][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:07:56,505][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:07:57,017][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:07:57,515][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:07:58,015][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:07:58,516][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:07:59,015][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:07:59,511][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:08:00,009][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:08:00,514][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:08:01,009][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:08:01,504][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:08:02,000][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:08:02,497][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:08:03,000][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:08:03,495][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:08:03,990][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:08:04,502][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:08:04,998][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:08:05,492][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:08:05,990][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:08:06,485][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:08:06,993][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:08:07,490][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:08:07,987][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:08:08,505][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:08:09,001][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:08:09,500][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:08:10,003][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:08:10,499][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:08:11,007][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:08:11,502][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:08:11,998][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:08:12,513][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:08:13,012][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:08:13,514][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:08:14,010][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:08:14,506][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:08:15,014][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:08:15,511][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:08:16,005][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:08:16,500][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:08:16,996][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:08:17,500][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:08:17,995][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:08:18,490][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:08:19,004][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:08:19,499][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:08:19,993][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:08:20,491][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:08:20,989][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:08:21,498][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:08:21,994][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:08:22,492][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:08:22,993][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:08:23,489][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:08:23,986][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:08:24,484][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:08:24,982][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:08:25,504][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:08:26,000][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:08:26,496][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:08:27,011][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:08:27,508][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:08:28,002][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10129 tokens.
[2025-11-12 23:08:28,655][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.01%, ΔTime: 00:00:32
[2025-11-12 23:08:29,438][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:08:29,439][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:08:29,441][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:08:30,399][__main__][INFO] - Iteration 66 took 51s (30.18% Gen, 67.96% Train). Generation: 15s, Training: 35s. Estimated remaining time: 42h 2m 45s. Estimated total time: 43h 1m 33s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 3s, 500 more iterations: 7h 10m 15s.
[2025-11-12 23:08:30,401][__main__][INFO] - Starting iteration 66.
[2025-11-12 23:08:30,910][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:08:30,910][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:08:32,999][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:08:40,670][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given the values, I recognize that hats are highly valued by me and moderately valued by Bob. Books are moderately valued by me and highly valued by Bob, and balls are valued similarly by both of us. To maximize my points, I will focus on claiming as many hats as possible since they are the most valuable items to me and I have a pretty clear advantage in this area. By proposing to take all 10 hats, I can secure a high value for myself while minimizing the risk of having the items split proportionally if my proposal exceeds the item quantity. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:08:47,484][__main__][INFO] - Number of regex retries in iteration 66: 2
[2025-11-12 23:08:47,485][__main__][INFO] - agents played in iteration 66 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:08:48,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:08:48,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:08:48,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:08:48,428][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:08:48,429][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:08:48,430][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:08:49,087][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:08:49,539][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:08:50,042][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:08:50,541][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:08:51,041][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:08:51,536][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:08:52,029][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:08:52,522][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:08:53,021][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:08:53,521][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:08:54,022][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:08:54,518][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:08:55,016][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:08:55,513][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:08:56,088][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:08:56,586][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:08:57,084][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:08:57,578][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:08:58,077][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:08:58,586][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:08:59,087][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:08:59,608][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:09:00,104][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:09:00,607][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:09:01,108][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:09:01,608][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:09:02,113][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:09:02,611][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:09:03,106][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:09:03,604][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:09:04,098][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:09:04,594][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:09:05,095][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:09:05,593][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:09:06,107][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:09:06,600][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:09:07,098][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:09:07,598][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:09:08,095][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:09:08,604][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:09:09,101][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:09:09,608][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:09:10,103][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:09:10,600][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:09:11,097][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:09:11,593][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:09:12,092][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:09:12,592][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:09:13,089][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:09:13,582][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:09:14,078][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:09:14,575][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:09:15,071][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:09:15,564][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:09:16,057][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:09:16,552][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:09:17,050][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:09:17,546][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:09:18,039][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:09:18,537][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:09:19,033][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:09:19,529][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:09:20,026][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:09:20,522][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:09:21,019][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10341 tokens.
[2025-11-12 23:09:21,673][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-12 23:09:22,433][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:09:22,435][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:09:22,437][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:09:23,358][__main__][INFO] - Iteration 67 took 52s (31.60% Gen, 66.64% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 42m 45s. Estimated total time: 43h 42m 26s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 24s, 500 more iterations: 7h 17m 4s.
[2025-11-12 23:09:23,361][__main__][INFO] - Starting iteration 67.
[2025-11-12 23:09:23,886][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:09:23,887][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:09:25,736][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:09:39,449][__main__][INFO] - Number of regex retries in iteration 67: 1
[2025-11-12 23:09:39,450][__main__][INFO] - agents played in iteration 67 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:09:40,323][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:09:40,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:09:40,371][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:09:40,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:09:40,393][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:09:40,394][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:09:41,022][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:09:41,484][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:09:41,989][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:09:42,485][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:09:42,989][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:09:43,484][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:09:43,981][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:09:44,477][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:09:44,975][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:09:45,477][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:09:45,979][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:09:46,477][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:09:46,983][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:09:47,479][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:09:47,978][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:09:48,475][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:09:48,974][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:09:49,472][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:09:49,970][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:09:50,466][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:09:50,965][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:09:51,461][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:09:51,954][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:09:52,450][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:09:52,946][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:09:53,446][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:09:53,943][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:09:54,439][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:09:54,935][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:09:55,430][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:09:55,929][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:09:56,429][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:09:56,928][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:09:57,435][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:09:57,932][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:09:58,431][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:09:58,930][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:09:59,429][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:09:59,930][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:10:00,425][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:10:00,922][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:10:01,429][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:10:01,923][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:10:02,422][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:10:02,915][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:10:03,410][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:10:03,919][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:10:04,414][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:10:04,911][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:10:05,413][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:10:05,911][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:10:06,421][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:10:06,924][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:10:07,421][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:10:07,930][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:10:08,425][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:10:08,919][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:10:09,418][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:10:09,915][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:10:10,420][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:10:10,915][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:10:11,411][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:10:11,910][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:10:12,428][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:10:12,935][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10219 tokens.
[2025-11-12 23:10:13,586][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.11%, ΔTime: 00:00:32
[2025-11-12 23:10:14,375][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:10:14,377][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:10:14,379][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:10:15,360][__main__][INFO] - Iteration 68 took 51s (30.21% Gen, 67.79% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 55m 33s. Estimated total time: 42h 56m 6s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 52s, 500 more iterations: 7h 9m 21s.
[2025-11-12 23:10:15,362][__main__][INFO] - Starting iteration 68.
[2025-11-12 23:10:15,845][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:10:15,845][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:10:31,369][__main__][INFO] - Number of regex retries in iteration 68: 0
[2025-11-12 23:10:31,370][__main__][INFO] - agents played in iteration 68 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:10:32,161][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:10:32,184][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:10:32,206][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:10:32,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:10:32,229][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:10:32,229][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:10:32,891][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:10:33,348][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:10:33,854][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:10:34,350][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:10:34,850][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:10:35,346][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:10:35,842][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:10:36,338][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:10:36,833][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:10:37,331][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:10:37,842][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:10:38,338][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:10:38,833][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:10:39,332][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:10:39,830][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:10:40,325][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:10:40,824][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:10:41,319][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:10:41,830][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:10:42,333][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:10:42,830][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:10:43,330][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:10:43,829][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:10:44,335][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:10:44,832][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:10:45,330][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:10:45,831][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:10:46,327][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:10:46,824][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:10:47,320][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:10:47,816][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:10:48,330][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:10:48,826][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:10:49,320][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:10:49,818][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:10:50,315][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:10:50,811][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:10:51,306][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:10:51,799][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:10:52,300][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:10:52,795][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:10:53,288][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:10:53,787][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:10:54,281][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:10:54,776][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:10:55,273][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:10:55,768][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:10:56,267][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:10:56,761][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:10:57,254][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:10:57,749][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:10:58,242][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:10:58,735][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:10:59,231][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:10:59,726][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:11:00,235][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:11:00,731][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:11:01,237][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:11:01,745][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:11:02,245][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:11:02,747][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:11:03,243][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:11:03,741][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:11:04,245][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:11:04,743][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10377 tokens.
[2025-11-12 23:11:05,420][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-12 23:11:06,181][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:11:06,183][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:11:06,185][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:11:07,114][__main__][INFO] - Iteration 69 took 51s (30.28% Gen, 67.90% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 42m 5s. Estimated total time: 42h 43m 31s. Time estimates for 10 more iterations: 8m 32s, 100 more iterations: 1h 25m 27s, 500 more iterations: 7h 7m 15s.
[2025-11-12 23:11:07,116][__main__][INFO] - Starting iteration 69.
[2025-11-12 23:11:07,598][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:11:07,599][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:11:11,616][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:11:23,035][__main__][INFO] - Number of regex retries in iteration 69: 1
[2025-11-12 23:11:23,036][__main__][INFO] - agents played in iteration 69 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:11:23,839][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:11:23,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:11:23,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:11:23,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:11:23,907][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:11:23,907][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:11:24,557][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:11:25,024][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:11:25,526][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:11:26,020][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:11:26,541][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:11:27,038][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:11:27,534][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:11:28,030][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:11:28,529][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:11:29,035][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:11:29,534][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:11:30,032][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:11:30,532][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:11:31,030][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:11:31,525][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:11:32,020][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:11:32,517][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:11:33,032][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:11:33,530][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:11:34,027][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:11:34,527][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:11:35,029][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:11:35,537][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:11:36,036][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:11:36,542][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:11:37,038][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:11:37,532][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:11:38,030][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:11:38,535][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:11:39,036][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:11:39,532][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:11:40,025][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:11:40,523][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:11:41,031][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:11:41,523][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:11:42,020][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:11:42,521][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:11:43,014][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:11:43,506][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:11:44,000][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:11:44,494][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:11:44,995][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:11:45,494][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:11:46,012][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:11:46,512][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:11:47,011][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:11:47,529][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:11:48,027][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:11:48,525][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:11:49,028][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:11:49,524][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:11:50,031][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:11:50,527][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:11:51,024][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:11:51,545][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:11:52,043][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:11:52,540][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:11:53,039][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:11:53,536][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:11:54,044][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:11:54,544][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:11:55,040][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:11:55,551][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:11:56,047][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:11:56,544][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10329 tokens.
[2025-11-12 23:11:57,219][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-12 23:11:57,987][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:11:57,988][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:11:57,990][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:11:58,888][__main__][INFO] - Iteration 70 took 51s (30.10% Gen, 68.15% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 42m 13s. Estimated total time: 42h 44m 30s. Time estimates for 10 more iterations: 8m 32s, 100 more iterations: 1h 25m 29s, 500 more iterations: 7h 7m 25s.
[2025-11-12 23:11:58,890][__main__][INFO] - Starting iteration 70.
[2025-11-12 23:11:59,395][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-12 23:11:59,396][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:12:01,424][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 5 hats, 5 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:12:14,850][__main__][INFO] - Number of regex retries in iteration 70: 1
[2025-11-12 23:12:14,850][__main__][INFO] - agents played in iteration 70 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:12:15,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:12:15,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:12:15,819][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:12:15,842][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:12:15,843][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:12:15,843][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:12:16,520][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:12:16,980][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:12:17,484][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:12:17,990][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:12:18,489][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:12:18,989][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:12:19,489][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:12:19,984][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:12:20,483][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:12:20,980][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:12:21,478][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:12:21,974][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:12:22,471][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:12:22,972][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:12:23,468][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:12:23,968][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:12:24,464][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:12:24,959][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:12:25,460][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:12:25,957][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:12:26,453][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:12:26,960][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:12:27,460][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:12:27,959][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:12:28,459][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:12:28,957][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:12:29,458][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:12:29,955][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:12:30,454][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:12:30,953][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:12:31,453][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:12:31,951][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:12:32,450][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:12:32,950][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:12:33,452][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:12:33,952][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:12:34,449][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:12:34,948][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:12:35,441][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:12:35,934][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:12:36,433][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:12:36,929][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:12:37,446][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:12:37,940][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:12:38,434][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:12:38,950][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:12:39,448][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:12:39,941][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:12:40,437][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:12:40,931][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:12:41,436][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:12:41,930][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:12:42,424][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:12:42,930][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:12:43,423][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:12:43,916][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:12:44,410][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:12:44,905][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:12:45,403][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:12:45,897][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:12:46,393][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:12:46,887][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:12:47,385][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:12:47,883][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:12:48,378][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10143 tokens.
[2025-11-12 23:12:49,079][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-12 23:12:49,846][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:12:49,848][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:12:49,850][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:12:51,719][__main__][INFO] - Iteration 71 took 52s (29.54% Gen, 66.89% Train). Generation: 15s, Training: 34s. Estimated remaining time: 42h 33m 2s. Estimated total time: 43h 36m 12s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 12s, 500 more iterations: 7h 16m 2s.
[2025-11-12 23:12:51,721][__main__][INFO] - Starting iteration 71.
[2025-11-12 23:12:52,258][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:12:52,259][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:13:08,657][__main__][INFO] - Number of regex retries in iteration 71: 0
[2025-11-12 23:13:08,657][__main__][INFO] - agents played in iteration 71 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:13:09,618][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:13:09,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:13:09,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:13:09,693][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:13:09,694][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:13:09,695][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:13:10,377][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:13:10,831][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:13:11,335][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:13:11,834][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:13:12,332][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:13:12,831][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:13:13,342][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:13:13,841][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:13:14,356][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:13:14,854][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:13:15,348][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:13:15,851][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:13:16,348][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:13:16,845][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:13:17,343][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:13:17,841][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:13:18,335][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:13:18,831][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:13:19,326][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:13:19,820][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:13:20,317][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:13:20,812][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:13:21,311][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:13:21,810][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:13:22,306][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:13:22,804][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:13:23,306][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:13:23,803][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:13:24,305][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:13:24,803][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:13:25,299][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:13:25,795][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:13:26,292][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:13:26,787][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:13:27,285][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:13:27,786][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:13:28,280][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:13:28,779][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:13:29,277][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:13:29,775][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:13:30,270][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:13:30,766][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:13:31,272][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:13:31,770][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:13:32,268][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:13:32,764][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:13:33,260][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:13:33,759][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:13:34,253][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:13:34,752][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:13:35,261][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:13:35,759][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:13:36,256][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:13:36,753][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:13:37,250][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:13:37,762][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:13:38,261][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:13:38,758][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:13:39,271][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:13:39,772][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:13:40,273][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:13:40,774][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:13:41,273][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:13:41,778][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:13:42,274][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10285 tokens.
[2025-11-12 23:13:42,927][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-12 23:13:43,716][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:13:43,717][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:13:43,719][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:13:44,838][__main__][INFO] - Iteration 72 took 52s (31.19% Gen, 66.68% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 44m 57s. Estimated total time: 43h 49m 0s. Time estimates for 10 more iterations: 8m 45s, 100 more iterations: 1h 27m 38s, 500 more iterations: 7h 18m 10s.
[2025-11-12 23:13:44,840][__main__][INFO] - Starting iteration 72.
[2025-11-12 23:13:45,411][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:13:45,412][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:14:01,499][__main__][INFO] - Number of regex retries in iteration 72: 0
[2025-11-12 23:14:01,500][__main__][INFO] - agents played in iteration 72 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:14:02,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:14:02,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:14:02,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:14:02,462][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:14:02,462][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:14:02,463][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:14:03,087][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:14:03,537][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:14:04,052][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:14:04,555][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:14:05,053][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:14:05,551][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:14:06,049][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:14:06,561][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:14:07,058][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:14:07,555][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:14:08,055][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:14:08,552][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:14:09,052][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:14:09,554][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:14:10,048][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:14:10,557][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:14:11,056][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:14:11,557][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:14:12,065][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:14:12,564][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:14:13,072][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:14:13,572][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:14:14,075][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:14:14,580][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:14:15,080][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:14:15,580][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:14:16,085][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:14:16,586][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:14:17,093][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:14:17,595][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:14:18,098][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:14:18,595][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:14:19,093][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:14:19,595][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:14:20,095][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:14:20,596][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:14:21,121][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:14:21,620][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:14:22,121][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:14:22,620][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:14:23,118][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:14:23,613][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:14:24,110][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:14:24,608][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:14:25,109][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:14:25,605][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:14:26,105][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:14:26,601][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:14:27,100][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:14:27,600][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:14:28,100][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:14:28,596][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:14:29,096][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:14:29,595][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:14:30,094][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:14:30,593][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:14:31,091][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:14:31,594][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:14:32,094][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:14:32,594][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:14:33,099][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:14:33,603][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:14:34,107][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:14:34,608][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:14:35,113][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10311 tokens.
[2025-11-12 23:14:35,809][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:32
[2025-11-12 23:14:36,595][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:14:36,598][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:14:36,601][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:14:37,597][__main__][INFO] - Iteration 73 took 52s (30.83% Gen, 67.26% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 24m 22s. Estimated total time: 43h 29m 18s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 58s, 500 more iterations: 7h 14m 53s.
[2025-11-12 23:14:37,599][__main__][INFO] - Starting iteration 73.
[2025-11-12 23:14:38,090][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:14:38,091][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:14:53,574][__main__][INFO] - Number of regex retries in iteration 73: 0
[2025-11-12 23:14:53,575][__main__][INFO] - agents played in iteration 73 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:14:54,364][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:14:54,389][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:14:54,412][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:14:54,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:14:54,434][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:14:54,435][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:14:55,096][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:14:55,668][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:14:56,131][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:14:56,632][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:14:57,127][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:14:57,625][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:14:58,120][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:14:58,615][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:14:59,111][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:14:59,607][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:15:00,102][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:15:00,604][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:15:01,097][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:15:01,593][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:15:02,090][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:15:02,585][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:15:03,089][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:15:03,587][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:15:04,084][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:15:04,584][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:15:05,086][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:15:05,581][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:15:06,080][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:15:06,577][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:15:07,077][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:15:07,578][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:15:08,078][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:15:08,579][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:15:09,079][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:15:09,575][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:15:10,073][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:15:10,572][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:15:11,087][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:15:11,583][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:15:12,080][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:15:12,576][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:15:13,073][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:15:13,575][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:15:14,077][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:15:14,572][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:15:15,069][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:15:15,565][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:15:16,061][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:15:16,571][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:15:17,068][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:15:17,579][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:15:18,075][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:15:18,573][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:15:19,074][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:15:19,574][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:15:20,086][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:15:20,587][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:15:21,085][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:15:21,598][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:15:22,096][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:15:22,595][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:15:23,090][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:15:23,589][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:15:24,100][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:15:24,600][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:15:25,098][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:15:25,593][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:15:26,092][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:15:26,599][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:15:27,100][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10295 tokens.
[2025-11-12 23:15:27,793][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.11%, ΔTime: 00:00:32
[2025-11-12 23:15:28,569][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:15:28,570][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:15:28,572][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:15:29,557][__main__][INFO] - Iteration 74 took 51s (30.08% Gen, 68.00% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 47m 34s. Estimated total time: 42h 53m 22s. Time estimates for 10 more iterations: 8m 34s, 100 more iterations: 1h 25m 46s, 500 more iterations: 7h 8m 53s.
[2025-11-12 23:15:29,559][__main__][INFO] - Starting iteration 74.
[2025-11-12 23:15:30,045][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:15:30,046][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:15:46,560][__main__][INFO] - Number of regex retries in iteration 74: 0
[2025-11-12 23:15:46,561][__main__][INFO] - agents played in iteration 74 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:15:47,566][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:15:47,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:15:47,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:15:47,637][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:15:47,638][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:15:47,639][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:15:48,306][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:15:48,780][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:15:49,285][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:15:49,785][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:15:50,290][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:15:50,791][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:15:51,290][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:15:51,792][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:15:52,292][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:15:52,796][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:15:53,293][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:15:53,787][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:15:54,286][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:15:54,783][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:15:55,281][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:15:55,781][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:15:56,279][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:15:56,806][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:15:57,313][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:15:57,818][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:15:58,327][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:15:58,835][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:15:59,335][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:15:59,834][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:16:00,333][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:16:00,834][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:16:01,337][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:16:01,836][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:16:02,336][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:16:02,834][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:16:03,336][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:16:03,833][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:16:04,331][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:16:04,824][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:16:05,318][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:16:05,813][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:16:06,309][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:16:06,802][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:16:07,309][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:16:07,804][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:16:08,298][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:16:08,794][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:16:09,291][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:16:09,793][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:16:10,292][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:16:10,788][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:16:11,303][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:16:11,797][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:16:12,294][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:16:12,790][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:16:13,287][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:16:13,796][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:16:14,293][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:16:14,787][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:16:15,286][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:16:15,783][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:16:16,279][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:16:16,779][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:16:17,273][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:16:17,791][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:16:18,284][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:16:18,778][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:16:19,280][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:16:19,777][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:16:20,276][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10430 tokens.
[2025-11-12 23:16:20,918][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-12 23:16:21,704][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:16:21,706][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:16:21,707][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:16:22,663][__main__][INFO] - Iteration 75 took 52s (31.39% Gen, 66.80% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 44m 14s. Estimated total time: 43h 50m 55s. Time estimates for 10 more iterations: 8m 46s, 100 more iterations: 1h 27m 41s, 500 more iterations: 7h 18m 29s.
[2025-11-12 23:16:22,665][__main__][INFO] - Starting iteration 75.
[2025-11-12 23:16:23,176][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:16:23,176][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:16:26,496][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:16:38,535][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:16:39,555][__main__][INFO] - Number of regex retries in iteration 75: 2
[2025-11-12 23:16:39,556][__main__][INFO] - agents played in iteration 75 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:16:40,513][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:16:40,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:16:40,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:16:40,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:16:40,586][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:16:40,587][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:16:41,284][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:16:41,738][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:16:42,241][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:16:42,739][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:16:43,241][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:16:43,740][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:16:44,241][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:16:44,738][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:16:45,236][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:16:45,736][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:16:46,231][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:16:46,731][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:16:47,230][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:16:47,729][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:16:48,222][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:16:48,721][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:16:49,224][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:16:49,725][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:16:50,224][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:16:50,717][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:16:51,217][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:16:51,719][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:16:52,219][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:16:52,720][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:16:53,218][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:16:53,718][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:16:54,217][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:16:54,715][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:16:55,217][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:16:55,720][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:16:56,219][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:16:56,721][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:16:57,219][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:16:57,737][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:16:58,242][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:16:58,741][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:16:59,240][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:16:59,739][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:17:00,238][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:17:00,739][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:17:01,237][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:17:01,733][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:17:02,232][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:17:02,731][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:17:03,231][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:17:03,732][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:17:04,228][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:17:04,728][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:17:05,223][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:17:05,720][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:17:06,216][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:17:06,715][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:17:07,216][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:17:07,710][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:17:08,205][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:17:08,710][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:17:09,206][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:17:09,708][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:17:10,221][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:17:10,719][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:17:11,220][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:17:11,720][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:17:12,220][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:17:12,717][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:17:13,211][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10385 tokens.
[2025-11-12 23:17:13,873][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-12 23:17:14,659][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:17:14,660][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:17:14,662][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:17:15,615][__main__][INFO] - Iteration 76 took 52s (31.23% Gen, 66.95% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 34m 25s. Estimated total time: 43h 41m 59s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 23s, 500 more iterations: 7h 16m 59s.
[2025-11-12 23:17:15,617][__main__][INFO] - Starting iteration 76.
[2025-11-12 23:17:16,155][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:17:16,156][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:17:32,090][__main__][INFO] - Number of regex retries in iteration 76: 0
[2025-11-12 23:17:32,091][__main__][INFO] - agents played in iteration 76 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:17:32,934][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:17:32,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:17:32,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:17:33,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:17:33,013][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:17:33,014][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:17:33,671][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:17:34,123][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:17:34,626][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:17:35,125][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:17:35,860][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:17:36,393][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:17:36,894][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:17:37,389][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:17:37,890][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:17:38,394][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:17:38,904][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:17:39,400][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:17:39,898][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:17:40,414][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:17:40,912][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:17:41,413][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:17:41,911][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:17:42,408][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:17:42,915][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:17:43,414][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:17:43,915][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:17:44,421][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:17:44,923][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:17:45,426][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:17:45,922][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:17:46,424][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:17:46,928][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:17:47,425][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:17:47,927][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:17:48,425][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:17:48,924][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:17:49,431][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:17:49,928][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:17:50,422][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:17:50,919][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:17:51,414][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:17:51,911][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:17:52,405][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:17:52,899][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:17:53,416][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:17:53,911][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:17:54,405][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:17:54,902][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:17:55,396][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:17:55,902][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:17:56,400][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:17:56,895][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:17:57,392][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:17:57,888][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:17:58,387][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:17:58,893][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:17:59,389][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:17:59,903][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:18:00,400][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:18:00,897][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:18:01,399][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:18:01,898][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:18:02,403][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:18:02,897][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:18:03,395][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:18:03,896][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:18:04,394][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:18:04,889][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:18:05,383][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:18:05,880][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10438 tokens.
[2025-11-12 23:18:06,545][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.08%, ΔTime: 00:00:32
[2025-11-12 23:18:07,321][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:18:07,323][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:18:07,324][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:18:08,311][__main__][INFO] - Iteration 77 took 52s (30.55% Gen, 67.55% Train). Generation: 15s, Training: 35s. Estimated remaining time: 42h 19m 24s. Estimated total time: 43h 27m 50s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 55s, 500 more iterations: 7h 14m 38s.
[2025-11-12 23:18:08,314][__main__][INFO] - Starting iteration 77.
[2025-11-12 23:18:08,872][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:18:08,872][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:18:25,598][__main__][INFO] - Number of regex retries in iteration 77: 0
[2025-11-12 23:18:25,599][__main__][INFO] - agents played in iteration 77 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:18:26,451][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:18:26,476][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:18:26,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:18:26,523][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:18:26,524][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:18:26,525][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:18:27,145][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:18:27,605][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:18:28,111][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:18:28,610][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:18:29,112][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:18:29,610][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:18:30,116][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:18:30,615][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:18:31,113][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:18:31,628][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:18:32,124][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:18:32,618][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:18:33,114][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:18:33,610][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:18:34,118][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:18:34,616][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:18:35,113][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:18:35,615][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:18:36,115][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:18:36,622][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:18:37,121][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:18:37,619][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:18:38,120][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:18:38,620][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:18:39,121][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:18:39,617][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:18:40,117][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:18:40,627][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:18:41,124][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:18:41,618][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:18:42,113][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:18:42,607][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:18:43,101][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:18:43,598][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:18:44,092][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:18:44,599][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:18:45,096][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:18:45,593][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:18:46,102][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:18:46,609][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:18:47,119][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:18:47,614][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:18:48,108][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:18:48,621][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:18:49,119][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:18:49,614][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:18:50,111][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:18:50,609][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:18:51,120][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:18:51,617][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:18:52,114][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:18:52,632][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:18:53,130][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:18:53,629][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:18:54,126][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:18:54,623][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:18:55,134][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:18:55,632][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:18:56,130][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:18:56,625][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:18:57,117][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:18:57,624][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:18:58,118][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:18:58,613][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:18:59,121][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10348 tokens.
[2025-11-12 23:18:59,798][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-12 23:19:00,595][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:19:00,597][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:19:00,599][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:19:01,571][__main__][INFO] - Iteration 78 took 52s (31.74% Gen, 66.41% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 45m 39s. Estimated total time: 43h 54m 59s. Time estimates for 10 more iterations: 8m 46s, 100 more iterations: 1h 27m 49s, 500 more iterations: 7h 19m 9s.
[2025-11-12 23:19:01,573][__main__][INFO] - Starting iteration 78.
[2025-11-12 23:19:02,094][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:19:02,094][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:19:17,986][__main__][INFO] - Number of regex retries in iteration 78: 0
[2025-11-12 23:19:17,986][__main__][INFO] - agents played in iteration 78 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:19:18,966][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:19:18,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:19:19,013][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:19:19,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:19:19,036][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:19:19,036][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:19:19,692][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:19:20,149][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:19:20,653][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:19:21,155][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:19:21,659][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:19:22,160][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:19:22,664][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:19:23,163][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:19:23,663][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:19:24,159][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:19:24,659][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:19:25,161][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:19:25,660][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:19:26,164][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:19:26,667][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:19:27,168][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:19:27,675][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:19:28,179][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:19:28,680][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:19:29,184][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:19:29,682][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:19:30,183][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:19:30,684][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:19:31,181][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:19:31,682][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:19:32,182][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:19:32,680][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:19:33,182][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:19:33,679][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:19:34,175][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:19:34,671][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:19:35,171][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:19:35,667][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:19:36,164][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:19:36,661][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:19:37,157][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:19:37,654][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:19:38,150][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:19:38,646][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:19:39,140][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:19:39,642][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:19:40,139][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:19:40,641][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:19:41,135][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:19:41,633][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:19:42,140][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:19:42,637][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:19:43,132][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:19:43,626][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:19:44,125][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:19:44,630][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:19:45,128][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:19:45,625][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:19:46,119][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:19:46,614][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:19:47,111][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:19:47,611][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:19:48,109][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:19:48,622][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:19:49,119][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:19:49,616][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:19:50,113][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:19:50,606][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:19:51,099][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:19:51,595][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10377 tokens.
[2025-11-12 23:19:52,292][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.13%, ΔTime: 00:00:32
[2025-11-12 23:19:53,067][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:19:53,068][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:19:53,070][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:19:54,209][__main__][INFO] - Iteration 79 took 52s (30.49% Gen, 67.32% Train). Generation: 15s, Training: 35s. Estimated remaining time: 42h 15m 34s. Estimated total time: 43h 25m 46s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 51s, 500 more iterations: 7h 14m 17s.
[2025-11-12 23:19:54,211][__main__][INFO] - Starting iteration 79.
[2025-11-12 23:19:54,757][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:19:54,757][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:19:56,940][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 5 hats, 5 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:20:07,371][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:20:10,939][__main__][INFO] - Number of regex retries in iteration 79: 2
[2025-11-12 23:20:10,939][__main__][INFO] - agents played in iteration 79 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:20:11,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:20:11,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:20:11,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:20:11,855][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:20:11,856][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:20:11,857][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:20:12,481][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:20:12,931][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:20:13,431][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:20:13,950][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:20:14,444][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:20:14,945][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:20:15,444][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:20:15,939][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:20:16,444][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:20:16,942][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:20:17,441][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:20:17,936][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:20:18,433][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:20:18,932][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:20:19,442][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:20:19,942][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:20:20,450][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:20:20,948][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:20:21,445][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:20:21,957][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:20:22,459][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:20:22,961][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:20:23,456][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:20:23,955][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:20:24,459][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:20:24,957][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:20:25,453][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:20:25,951][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:20:26,447][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:20:26,949][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:20:27,448][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:20:27,945][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:20:28,443][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:20:28,936][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:20:29,432][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:20:29,932][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:20:30,426][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:20:30,925][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:20:31,419][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:20:31,912][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:20:32,408][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:20:32,902][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:20:33,397][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:20:33,892][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:20:34,390][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:20:34,892][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:20:35,389][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:20:35,885][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:20:36,383][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:20:36,879][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:20:37,378][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:20:37,876][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:20:38,370][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:20:38,870][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:20:39,365][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:20:39,862][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:20:40,359][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:20:40,857][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:20:41,358][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:20:41,854][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:20:42,352][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:20:42,856][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:20:43,353][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:20:43,851][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:20:44,350][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10341 tokens.
[2025-11-12 23:20:45,034][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-12 23:20:45,820][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:20:45,821][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:20:45,823][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:20:46,988][__main__][INFO] - Iteration 80 took 52s (30.98% Gen, 66.79% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 20m 30s. Estimated total time: 43h 31m 36s. Time estimates for 10 more iterations: 8m 42s, 100 more iterations: 1h 27m 3s, 500 more iterations: 7h 15m 16s.
[2025-11-12 23:20:46,993][__main__][INFO] - Starting iteration 80.
[2025-11-12 23:20:47,483][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-12 23:20:47,484][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:20:49,978][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:20:57,706][mllm.models.large_language_model_local][WARNING] - Response Given the new values, it seems Bob places a higher value on books and I place a higher value on balls. To maximize my points, I should keep the balls and let Bob have the books and hats. Here’s my proposal:

Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:21:04,157][__main__][INFO] - Number of regex retries in iteration 80: 2
[2025-11-12 23:21:04,158][__main__][INFO] - agents played in iteration 80 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:21:04,981][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:21:05,011][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:21:05,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:21:05,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:21:05,066][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:21:05,067][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:21:05,677][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:21:06,126][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:21:06,630][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:21:07,131][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:21:07,627][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:21:08,123][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:21:08,618][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:21:09,115][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:21:09,611][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:21:10,108][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:21:10,608][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:21:11,102][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:21:11,597][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:21:12,098][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:21:12,595][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:21:13,099][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:21:13,600][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:21:14,099][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:21:14,600][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:21:15,097][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:21:15,599][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:21:16,093][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:21:16,592][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:21:17,094][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:21:17,593][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:21:18,094][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:21:18,594][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:21:19,092][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:21:19,595][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:21:20,096][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:21:20,594][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:21:21,092][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:21:21,591][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:21:22,091][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:21:22,585][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:21:23,079][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:21:23,583][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:21:24,081][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:21:24,580][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:21:25,077][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:21:25,574][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:21:26,078][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:21:26,580][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:21:27,082][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:21:27,583][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:21:28,082][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:21:28,583][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:21:29,081][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:21:29,581][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:21:30,084][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:21:30,581][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:21:31,079][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:21:31,578][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:21:32,076][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:21:32,577][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:21:33,075][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:21:33,574][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:21:34,075][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:21:34,571][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:21:35,069][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:21:35,562][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:21:36,058][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:21:36,557][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:21:37,057][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:21:37,559][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10528 tokens.
[2025-11-12 23:21:38,237][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:32
[2025-11-12 23:21:39,013][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:21:39,015][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:21:39,016][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:21:40,979][__main__][INFO] - Iteration 81 took 53s (31.17% Gen, 65.16% Train). Generation: 16s, Training: 34s. Estimated remaining time: 43h 22m 48s. Estimated total time: 44h 34m 47s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 9s, 500 more iterations: 7h 25m 47s.
[2025-11-12 23:21:40,981][__main__][INFO] - Starting iteration 81.
[2025-11-12 23:21:41,492][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:21:41,492][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:21:47,143][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:21:58,705][__main__][INFO] - Number of regex retries in iteration 81: 1
[2025-11-12 23:21:58,706][__main__][INFO] - agents played in iteration 81 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:21:59,674][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:21:59,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:21:59,730][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:21:59,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:21:59,754][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:21:59,755][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:22:00,423][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:22:00,876][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:22:01,382][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:22:01,879][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:22:02,375][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:22:02,875][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:22:03,371][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:22:03,873][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:22:04,368][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:22:04,864][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:22:05,364][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:22:05,869][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:22:06,374][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:22:06,882][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:22:07,383][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:22:07,881][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:22:08,376][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:22:08,876][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:22:09,381][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:22:09,880][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:22:10,379][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:22:10,881][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:22:11,376][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:22:11,872][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:22:12,374][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:22:12,869][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:22:13,365][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:22:13,862][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:22:14,359][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:22:14,868][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:22:15,363][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:22:15,860][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:22:16,363][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:22:16,857][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:22:17,365][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:22:17,864][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:22:18,358][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:22:18,865][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:22:19,359][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:22:19,854][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:22:20,353][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:22:20,852][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:22:21,359][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:22:21,853][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:22:22,351][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:22:22,850][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:22:23,345][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:22:23,838][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:22:24,336][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:22:24,833][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:22:25,350][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:22:25,848][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:22:26,344][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:22:26,853][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:22:27,349][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:22:27,852][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:22:28,349][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:22:28,842][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:22:29,340][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:22:29,833][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:22:30,329][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:22:30,824][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:22:31,317][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:22:31,811][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:22:32,304][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10459 tokens.
[2025-11-12 23:22:32,961][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:32
[2025-11-12 23:22:33,729][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:22:33,731][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:22:33,732][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:22:34,662][__main__][INFO] - Iteration 82 took 53s (32.37% Gen, 65.87% Train). Generation: 17s, Training: 35s. Estimated remaining time: 43h 5m 40s. Estimated total time: 44h 18m 33s. Time estimates for 10 more iterations: 8m 51s, 100 more iterations: 1h 28m 37s, 500 more iterations: 7h 23m 5s.
[2025-11-12 23:22:34,664][__main__][INFO] - Starting iteration 82.
[2025-11-12 23:22:35,172][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:22:35,173][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:22:51,268][__main__][INFO] - Number of regex retries in iteration 82: 0
[2025-11-12 23:22:51,268][__main__][INFO] - agents played in iteration 82 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:22:52,215][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:22:52,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:22:52,259][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:22:52,281][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:22:52,282][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:22:52,283][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:22:52,936][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:22:53,388][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:22:53,890][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:22:54,393][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:22:54,888][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:22:55,384][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:22:55,980][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:22:56,480][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:22:56,978][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:22:57,474][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:22:57,975][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:22:58,475][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:22:58,973][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:22:59,489][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:22:59,988][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:23:00,486][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:23:00,985][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:23:01,486][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:23:01,992][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:23:02,491][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:23:02,989][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:23:03,496][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:23:03,992][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:23:04,490][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:23:04,985][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:23:05,480][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:23:05,994][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:23:06,499][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:23:06,999][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:23:07,495][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:23:07,996][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:23:08,494][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:23:08,992][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:23:09,489][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:23:09,998][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:23:10,515][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:23:11,011][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:23:11,514][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:23:12,010][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:23:12,509][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:23:13,009][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:23:13,511][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:23:14,012][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:23:14,512][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:23:15,016][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:23:15,518][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:23:16,017][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:23:16,520][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:23:17,019][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:23:17,518][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:23:18,018][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:23:18,516][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:23:19,015][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:23:19,511][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:23:20,011][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:23:20,514][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:23:21,011][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:23:21,504][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:23:22,001][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:23:22,499][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:23:23,000][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:23:23,498][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:23:23,997][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:23:24,504][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:23:25,000][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10424 tokens.
[2025-11-12 23:23:25,703][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-12 23:23:26,635][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:23:26,636][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:23:26,638][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:23:27,566][__main__][INFO] - Iteration 83 took 52s (30.72% Gen, 67.51% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 25m 57s. Estimated total time: 43h 39m 43s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 19s, 500 more iterations: 7h 16m 37s.
[2025-11-12 23:23:27,568][__main__][INFO] - Starting iteration 83.
[2025-11-12 23:23:28,122][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:23:28,123][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:23:30,357][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:23:43,961][__main__][INFO] - Number of regex retries in iteration 83: 1
[2025-11-12 23:23:43,961][__main__][INFO] - agents played in iteration 83 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:23:44,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:23:44,914][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:23:44,940][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:23:44,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:23:44,964][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:23:44,965][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:23:45,631][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:23:46,090][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:23:46,594][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:23:47,097][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:23:47,599][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:23:48,106][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:23:48,610][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:23:49,114][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:23:49,617][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:23:50,119][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:23:50,623][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:23:51,127][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:23:51,628][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:23:52,132][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:23:52,638][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:23:53,138][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:23:53,640][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:23:54,140][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:23:54,640][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:23:55,138][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:23:55,636][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:23:56,133][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:23:56,634][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:23:57,131][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:23:57,628][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:23:58,125][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:23:58,620][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:23:59,117][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:23:59,616][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:24:00,113][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:24:00,611][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:24:01,111][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:24:01,608][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:24:02,106][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:24:02,602][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:24:03,102][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:24:03,600][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:24:04,110][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:24:04,613][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:24:05,112][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:24:05,613][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:24:06,113][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:24:06,617][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:24:07,115][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:24:07,612][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:24:08,111][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:24:08,611][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:24:09,113][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:24:09,614][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:24:10,113][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:24:10,617][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:24:11,118][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:24:11,616][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:24:12,112][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:24:12,610][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:24:13,112][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:24:13,613][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:24:14,108][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:24:14,614][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:24:15,111][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:24:15,606][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:24:16,102][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:24:16,595][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:24:17,109][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:24:17,602][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10665 tokens.
[2025-11-12 23:24:18,327][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-12 23:24:19,109][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:24:19,110][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:24:19,112][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:24:20,077][__main__][INFO] - Iteration 84 took 51s (30.48% Gen, 67.66% Train). Generation: 15s, Training: 35s. Estimated remaining time: 42h 3m 8s. Estimated total time: 43h 17m 46s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 35s, 500 more iterations: 7h 12m 57s.
[2025-11-12 23:24:20,079][__main__][INFO] - Starting iteration 84.
[2025-11-12 23:24:20,644][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:24:20,644][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:24:28,990][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:24:31,321][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 10 books, 0 balls

Given the per-item values:
- You value hats at 1, books at 10, and balls at 10.
- Bob values hats at 10, books at 1, and balls at 10.

Since you value books and balls highly and Bob values them more than hats, and you both value these items equally, a reasonable strategy would be to propose taking all the books and balls, and all the hats. This ensures that you capture the items you highly value and minimizes the risk of an unequal split. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:24:38,411][__main__][INFO] - Number of regex retries in iteration 84: 2
[2025-11-12 23:24:38,412][__main__][INFO] - agents played in iteration 84 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:24:39,250][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:24:39,277][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:24:39,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:24:39,327][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:24:39,327][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:24:39,328][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:24:39,934][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:24:40,389][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:24:40,902][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:24:41,403][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:24:41,917][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:24:42,422][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:24:42,925][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:24:43,428][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:24:43,934][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:24:44,438][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:24:44,942][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:24:45,445][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:24:45,944][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:24:46,458][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:24:46,956][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:24:47,455][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:24:47,952][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:24:48,450][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:24:48,947][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:24:49,446][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:24:49,949][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:24:50,449][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:24:50,948][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:24:51,446][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:24:51,944][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:24:52,441][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:24:52,937][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:24:53,435][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:24:53,932][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:24:54,430][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:24:54,933][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:24:55,429][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:24:55,925][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:24:56,428][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:24:56,926][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:24:57,427][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:24:57,927][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:24:58,427][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:24:58,934][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:24:59,434][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:24:59,931][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:25:00,431][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:25:00,926][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:25:01,428][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:25:01,925][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:25:02,417][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:25:02,913][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:25:03,407][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:25:03,901][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:25:04,399][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:25:04,896][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:25:05,407][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:25:05,905][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:25:06,403][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:25:06,914][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:25:07,413][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:25:07,914][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:25:08,407][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:25:08,902][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:25:09,414][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:25:09,908][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:25:10,403][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:25:10,903][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:25:11,403][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:25:11,914][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10527 tokens.
[2025-11-12 23:25:12,620][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-12 23:25:13,424][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:25:13,427][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:25:13,428][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:25:14,452][__main__][INFO] - Iteration 85 took 53s (33.02% Gen, 65.08% Train). Generation: 17s, Training: 35s. Estimated remaining time: 43h 34m 53s. Estimated total time: 44h 50m 26s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 40s, 500 more iterations: 7h 28m 24s.
[2025-11-12 23:25:14,454][__main__][INFO] - Starting iteration 85.
[2025-11-12 23:25:14,997][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:25:14,998][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:25:22,264][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given the values, hats are highly valuable to me and fairly worthless to Alice. Books are valuable to Alice but not to me. Balls are of low value to both of us. By proposing to take all the hats and none of the other items, I aim to maximize my points from the hats, as they offer the highest value to me. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:25:24,307][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:25:32,434][__main__][INFO] - Number of regex retries in iteration 85: 2
[2025-11-12 23:25:32,435][__main__][INFO] - agents played in iteration 85 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:25:33,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:25:33,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:25:33,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:25:33,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:25:33,375][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:25:33,376][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:25:34,040][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:25:34,523][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:25:35,028][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:25:35,528][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:25:36,030][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:25:36,531][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:25:37,037][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:25:37,537][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:25:38,039][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:25:38,544][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:25:39,043][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:25:39,542][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:25:40,042][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:25:40,537][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:25:41,041][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:25:41,539][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:25:42,039][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:25:42,540][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:25:43,038][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:25:43,538][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:25:44,036][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:25:44,535][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:25:45,037][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:25:45,536][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:25:46,034][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:25:46,540][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:25:47,039][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:25:47,539][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:25:48,039][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:25:48,538][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:25:49,044][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:25:49,544][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:25:50,039][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:25:50,538][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:25:51,037][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:25:51,542][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:25:52,043][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:25:52,542][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:25:53,041][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:25:53,539][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:25:54,038][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:25:54,533][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:25:55,030][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:25:55,540][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:25:56,037][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:25:56,535][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:25:57,034][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:25:57,533][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:25:58,034][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:25:58,529][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:25:59,029][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:25:59,545][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:26:00,041][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:26:00,538][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:26:01,037][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:26:01,537][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:26:02,043][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:26:02,544][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:26:03,043][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:26:03,543][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:26:04,047][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:26:04,547][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:26:05,048][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:26:05,549][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:26:06,053][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10579 tokens.
[2025-11-12 23:26:06,704][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-12 23:26:07,495][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:26:07,497][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:26:07,498][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:26:08,435][__main__][INFO] - Iteration 86 took 53s (32.59% Gen, 65.54% Train). Generation: 17s, Training: 35s. Estimated remaining time: 43h 18m 37s. Estimated total time: 44h 35m 4s. Time estimates for 10 more iterations: 8m 55s, 100 more iterations: 1h 29m 10s, 500 more iterations: 7h 25m 50s.
[2025-11-12 23:26:08,437][__main__][INFO] - Starting iteration 86.
[2025-11-12 23:26:08,903][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:26:08,904][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:26:15,399][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:26:24,793][__main__][INFO] - Number of regex retries in iteration 86: 1
[2025-11-12 23:26:24,794][__main__][INFO] - agents played in iteration 86 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:26:25,605][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:26:25,634][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:26:25,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:26:25,698][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:26:25,699][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:26:25,700][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:26:26,370][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:26:26,839][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:26:27,345][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:26:27,846][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:26:28,352][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:26:28,853][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:26:29,357][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:26:29,864][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:26:30,365][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:26:30,864][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:26:31,361][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:26:31,858][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:26:32,358][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:26:32,856][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:26:33,355][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:26:33,858][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:26:34,356][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:26:34,856][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:26:35,353][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:26:35,851][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:26:36,360][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:26:36,861][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:26:37,359][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:26:37,864][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:26:38,361][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:26:38,862][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:26:39,362][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:26:39,860][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:26:40,364][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:26:40,862][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:26:41,363][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:26:41,863][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:26:42,365][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:26:42,867][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:26:43,365][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:26:43,865][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:26:44,370][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:26:44,870][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:26:45,369][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:26:45,870][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:26:46,370][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:26:46,868][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:26:47,362][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:26:47,861][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:26:48,354][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:26:48,853][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:26:49,356][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:26:49,853][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:26:50,352][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:26:50,850][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:26:51,348][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:26:51,847][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:26:52,345][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:26:52,842][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:26:53,356][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:26:53,854][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:26:54,352][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:26:54,856][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:26:55,355][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:26:55,859][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:26:56,357][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:26:56,854][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:26:57,358][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:26:57,857][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:26:58,365][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10730 tokens.
[2025-11-12 23:26:59,055][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-12 23:26:59,999][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:27:00,001][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:27:00,016][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:27:01,293][__main__][INFO] - Iteration 87 took 52s (30.33% Gen, 67.23% Train). Generation: 15s, Training: 35s. Estimated remaining time: 42h 22m 11s. Estimated total time: 43h 39m 30s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 19s, 500 more iterations: 7h 16m 35s.
[2025-11-12 23:27:01,295][__main__][INFO] - Starting iteration 87.
[2025-11-12 23:27:01,780][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:27:01,781][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:27:17,924][__main__][INFO] - Number of regex retries in iteration 87: 0
[2025-11-12 23:27:17,924][__main__][INFO] - agents played in iteration 87 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:27:18,728][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:27:18,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:27:18,773][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:27:18,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:27:18,796][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:27:18,796][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:27:19,442][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:27:19,899][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:27:20,405][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:27:20,905][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:27:21,405][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:27:21,908][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:27:22,423][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:27:22,923][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:27:23,426][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:27:23,920][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:27:24,415][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:27:24,910][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:27:25,409][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:27:25,907][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:27:26,407][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:27:26,902][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:27:27,401][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:27:27,897][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:27:28,395][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:27:28,895][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:27:29,391][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:27:29,887][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:27:30,386][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:27:30,886][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:27:31,387][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:27:31,883][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:27:32,381][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:27:32,877][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:27:33,373][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:27:33,873][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:27:34,372][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:27:34,870][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:27:35,370][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:27:35,866][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:27:36,361][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:27:36,857][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:27:37,355][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:27:37,852][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:27:38,351][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:27:38,847][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:27:39,345][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:27:39,841][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:27:40,350][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:27:40,846][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:27:41,343][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:27:41,841][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:27:42,336][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:27:42,832][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:27:43,334][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:27:43,828][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:27:44,345][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:27:44,841][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:27:45,338][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:27:45,839][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:27:46,335][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:27:46,846][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:27:47,346][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:27:47,845][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:27:48,348][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:27:48,847][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:27:49,365][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:27:49,865][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:27:50,362][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:27:50,858][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:27:51,355][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10548 tokens.
[2025-11-12 23:27:52,032][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-12 23:27:52,761][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:27:52,763][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:27:52,764][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:27:53,744][__main__][INFO] - Iteration 88 took 51s (31.07% Gen, 67.05% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 0m 1s. Estimated total time: 43h 18m 13s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 36s, 500 more iterations: 7h 13m 2s.
[2025-11-12 23:27:53,747][__main__][INFO] - Starting iteration 88.
[2025-11-12 23:27:54,227][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:27:54,228][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:28:10,400][__main__][INFO] - Number of regex retries in iteration 88: 0
[2025-11-12 23:28:10,401][__main__][INFO] - agents played in iteration 88 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:28:11,233][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:28:11,259][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:28:11,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:28:11,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:28:11,306][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:28:11,307][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:28:11,967][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:28:12,424][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:28:12,930][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:28:13,433][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:28:13,934][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:28:14,449][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:28:14,950][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:28:15,452][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:28:15,952][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:28:16,460][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:28:16,973][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:28:17,475][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:28:17,975][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:28:18,499][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:28:19,001][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:28:19,502][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:28:20,007][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:28:20,512][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:28:21,010][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:28:21,511][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:28:22,013][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:28:22,512][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:28:23,015][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:28:23,516][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:28:24,017][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:28:24,514][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:28:25,008][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:28:25,505][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:28:26,006][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:28:26,503][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:28:26,998][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:28:27,496][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:28:27,995][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:28:28,496][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:28:28,994][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:28:29,493][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:28:29,992][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:28:30,487][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:28:30,984][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:28:31,479][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:28:31,976][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:28:32,481][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:28:32,976][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:28:33,471][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:28:33,967][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:28:34,463][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:28:34,956][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:28:35,448][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:28:35,941][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:28:36,446][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:28:36,943][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:28:37,444][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:28:37,952][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:28:38,449][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:28:38,951][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:28:39,447][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:28:39,943][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:28:40,454][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:28:40,951][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:28:41,453][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:28:41,954][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:28:42,455][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:28:42,974][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:28:43,474][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:28:43,975][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10686 tokens.
[2025-11-12 23:28:44,643][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-12 23:28:45,368][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:28:45,371][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:28:45,373][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:28:46,290][__main__][INFO] - Iteration 89 took 52s (31.06% Gen, 67.17% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 4m 6s. Estimated total time: 43h 23m 10s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 46s, 500 more iterations: 7h 13m 51s.
[2025-11-12 23:28:46,293][__main__][INFO] - Starting iteration 89.
[2025-11-12 23:28:46,776][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:28:46,777][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:28:50,579][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:29:02,741][__main__][INFO] - Number of regex retries in iteration 89: 1
[2025-11-12 23:29:02,742][__main__][INFO] - agents played in iteration 89 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:29:03,562][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:29:03,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:29:03,619][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:29:03,644][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:29:03,645][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:29:03,647][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:29:04,274][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:29:04,730][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:29:05,232][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:29:05,732][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:29:06,230][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:29:06,733][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:29:07,233][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:29:07,734][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:29:08,236][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:29:08,736][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:29:09,239][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:29:09,740][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:29:10,239][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:29:10,739][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:29:11,241][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:29:11,745][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:29:12,259][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:29:12,759][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:29:13,259][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:29:13,761][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:29:14,261][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:29:14,771][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:29:15,270][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:29:15,765][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:29:16,266][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:29:16,764][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:29:17,259][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:29:17,757][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:29:18,254][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:29:18,776][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:29:19,274][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:29:19,768][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:29:20,269][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:29:20,766][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:29:21,272][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:29:21,766][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:29:22,264][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:29:22,764][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:29:23,262][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:29:23,762][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:29:24,259][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:29:24,757][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:29:25,271][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:29:25,771][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:29:26,270][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:29:26,784][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:29:27,284][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:29:27,783][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:29:28,283][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:29:28,778][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:29:29,274][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:29:29,772][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:29:30,267][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:29:30,762][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:29:31,257][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:29:31,754][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:29:32,264][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:29:32,762][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:29:33,284][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:29:33,785][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:29:34,287][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:29:34,797][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:29:35,300][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:29:35,800][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:29:36,298][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10725 tokens.
[2025-11-12 23:29:36,965][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-12 23:29:37,893][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:29:37,895][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:29:37,912][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:29:39,030][__main__][INFO] - Iteration 90 took 52s (30.55% Gen, 67.31% Train). Generation: 15s, Training: 35s. Estimated remaining time: 42h 12m 44s. Estimated total time: 43h 32m 41s. Time estimates for 10 more iterations: 8m 42s, 100 more iterations: 1h 27m 5s, 500 more iterations: 7h 15m 26s.
[2025-11-12 23:29:39,032][__main__][INFO] - Starting iteration 90.
[2025-11-12 23:29:39,513][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-12 23:29:39,514][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:29:41,928][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:29:54,087][__main__][INFO] - Number of regex retries in iteration 90: 1
[2025-11-12 23:29:54,087][__main__][INFO] - agents played in iteration 90 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:29:54,900][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:29:54,928][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:29:54,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:29:54,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:29:54,978][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:29:54,978][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:29:55,621][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:29:56,083][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:29:56,588][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:29:57,102][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:29:57,602][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:29:58,105][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:29:58,605][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:29:59,106][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:29:59,610][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:30:00,110][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:30:00,612][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:30:01,115][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:30:01,617][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:30:02,117][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:30:02,614][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:30:03,115][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:30:03,616][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:30:04,115][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:30:04,619][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:30:05,122][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:30:05,629][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:30:06,130][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:30:06,633][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:30:07,133][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:30:07,634][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:30:08,129][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:30:08,624][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:30:09,120][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:30:09,619][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:30:10,118][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:30:10,614][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:30:11,109][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:30:11,605][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:30:12,107][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:30:12,611][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:30:13,112][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:30:13,613][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:30:14,116][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:30:14,614][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:30:15,115][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:30:15,615][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:30:16,115][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:30:16,613][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:30:17,112][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:30:17,611][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:30:18,113][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:30:18,612][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:30:19,112][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:30:19,609][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:30:20,104][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:30:20,604][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:30:21,103][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:30:21,601][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:30:22,099][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:30:22,598][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:30:23,097][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:30:23,594][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:30:24,088][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:30:24,603][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:30:25,098][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:30:25,594][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:30:26,092][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:30:26,589][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:30:27,096][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:30:27,595][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10592 tokens.
[2025-11-12 23:30:28,277][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-12 23:30:29,028][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:30:29,030][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:30:29,032][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:30:31,071][__main__][INFO] - Iteration 91 took 51s (28.26% Gen, 67.78% Train). Generation: 14s, Training: 34s. Estimated remaining time: 41h 37m 8s. Estimated total time: 42h 57m 58s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 55s, 500 more iterations: 7h 9m 39s.
[2025-11-12 23:30:31,073][__main__][INFO] - Starting iteration 91.
[2025-11-12 23:30:31,581][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:30:31,582][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:30:34,542][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:30:34,626][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:30:48,387][__main__][INFO] - Number of regex retries in iteration 91: 2
[2025-11-12 23:30:48,388][__main__][INFO] - agents played in iteration 91 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:30:49,249][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:30:49,277][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:30:49,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:30:49,327][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:30:49,327][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:30:49,328][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:30:49,970][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:30:50,425][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:30:50,928][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:30:51,427][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:30:51,926][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:30:52,425][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:30:52,926][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:30:53,427][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:30:53,928][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:30:54,432][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:30:54,936][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:30:55,437][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:30:55,935][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:30:56,436][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:30:56,946][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:30:57,451][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:30:57,953][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:30:58,456][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:30:58,959][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:30:59,458][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:30:59,957][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:31:00,457][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:31:00,955][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:31:01,456][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:31:01,954][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:31:02,455][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:31:02,955][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:31:03,457][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:31:03,963][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:31:04,461][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:31:04,959][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:31:05,455][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:31:05,955][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:31:06,458][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:31:06,957][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:31:07,457][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:31:07,956][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:31:08,457][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:31:08,958][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:31:09,457][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:31:09,957][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:31:10,457][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:31:10,956][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:31:11,454][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:31:11,953][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:31:12,453][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:31:12,952][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:31:13,454][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:31:13,950][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:31:14,449][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:31:14,945][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:31:15,474][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:31:15,973][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:31:16,477][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:31:16,992][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:31:17,492][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:31:17,992][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:31:18,491][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:31:18,990][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:31:19,487][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:31:19,986][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:31:20,488][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:31:20,990][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:31:21,487][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:31:21,986][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10712 tokens.
[2025-11-12 23:31:22,674][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-12 23:31:23,458][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:31:23,459][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:31:23,461][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:31:24,349][__main__][INFO] - Iteration 92 took 52s (31.85% Gen, 66.46% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 36m 45s. Estimated total time: 43h 58m 28s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 56s, 500 more iterations: 7h 19m 44s.
[2025-11-12 23:31:24,351][__main__][INFO] - Starting iteration 92.
[2025-11-12 23:31:24,878][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:31:24,879][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:31:40,547][__main__][INFO] - Number of regex retries in iteration 92: 0
[2025-11-12 23:31:40,548][__main__][INFO] - agents played in iteration 92 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:31:41,399][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:31:41,428][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:31:41,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:31:41,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:31:41,479][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:31:41,480][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:31:42,110][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:31:42,565][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:31:43,073][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:31:43,587][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:31:44,087][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:31:44,592][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:31:45,093][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:31:45,598][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:31:46,118][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:31:46,620][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:31:47,121][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:31:47,619][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:31:48,121][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:31:48,625][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:31:49,124][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:31:49,621][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:31:50,122][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:31:50,616][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:31:51,112][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:31:51,610][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:31:52,106][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:31:52,612][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:31:53,110][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:31:53,607][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:31:54,123][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:31:54,620][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:31:55,116][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:31:55,615][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:31:56,115][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:31:56,627][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:31:57,123][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:31:57,619][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:31:58,117][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:31:58,616][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:31:59,111][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:31:59,605][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:32:00,103][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:32:00,616][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:32:01,114][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:32:01,611][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:32:02,106][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:32:02,605][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:32:03,112][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:32:03,612][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:32:04,109][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:32:04,609][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:32:05,107][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:32:05,606][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:32:06,104][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:32:06,598][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:32:07,112][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:32:07,613][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:32:08,105][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:32:08,604][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:32:09,100][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:32:09,609][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:32:10,105][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:32:10,603][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:32:11,115][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:32:11,612][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:32:12,109][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:32:12,609][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:32:13,106][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:32:13,616][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:32:14,116][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10802 tokens.
[2025-11-12 23:32:14,781][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-12 23:32:15,549][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:32:15,551][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:32:15,553][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:32:16,511][__main__][INFO] - Iteration 93 took 51s (30.32% Gen, 67.74% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 41m 17s. Estimated total time: 43h 3m 52s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 7s, 500 more iterations: 7h 10m 38s.
[2025-11-12 23:32:16,513][__main__][INFO] - Starting iteration 93.
[2025-11-12 23:32:16,984][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:32:16,984][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:32:32,023][__main__][INFO] - Number of regex retries in iteration 93: 0
[2025-11-12 23:32:32,024][__main__][INFO] - agents played in iteration 93 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:32:32,817][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:32:32,840][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:32:32,865][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:32:32,888][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:32:32,889][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:32:32,890][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:32:33,528][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:32:33,980][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:32:34,486][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:32:34,988][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:32:35,487][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:32:35,991][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:32:36,490][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:32:36,990][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:32:37,495][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:32:38,001][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:32:38,508][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:32:39,009][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:32:39,513][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:32:40,028][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:32:40,527][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:32:41,026][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:32:41,524][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:32:42,019][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:32:42,528][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:32:43,024][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:32:43,519][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:32:44,018][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:32:44,515][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:32:45,009][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:32:45,505][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:32:46,003][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:32:46,529][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:32:47,029][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:32:47,527][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:32:48,033][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:32:48,530][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:32:49,027][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:32:49,526][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:32:50,019][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:32:50,514][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:32:51,009][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:32:51,504][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:32:52,015][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:32:52,509][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:32:53,002][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:32:53,497][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:32:53,994][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:32:54,493][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:32:54,989][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:32:55,486][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:32:55,985][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:32:56,482][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:32:56,979][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:32:57,476][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:32:57,970][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:32:58,472][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:32:58,969][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:32:59,466][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:32:59,963][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:33:00,457][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:33:00,955][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:33:01,452][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:33:01,947][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:33:02,446][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:33:02,942][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:33:03,435][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:33:03,945][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:33:04,443][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:33:04,941][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:33:05,436][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10828 tokens.
[2025-11-12 23:33:06,105][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-12 23:33:06,898][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:33:06,899][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:33:06,901][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:33:07,832][__main__][INFO] - Iteration 94 took 50s (29.58% Gen, 68.59% Train). Generation: 15s, Training: 34s. Estimated remaining time: 40h 59m 1s. Estimated total time: 42h 22m 27s. Time estimates for 10 more iterations: 8m 28s, 100 more iterations: 1h 24m 44s, 500 more iterations: 7h 3m 44s.
[2025-11-12 23:33:07,835][__main__][INFO] - Starting iteration 94.
[2025-11-12 23:33:08,364][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:33:08,365][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:33:23,794][__main__][INFO] - Number of regex retries in iteration 94: 0
[2025-11-12 23:33:23,795][__main__][INFO] - agents played in iteration 94 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:33:24,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:33:24,747][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:33:24,773][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:33:24,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:33:24,796][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:33:24,796][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:33:25,462][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:33:25,920][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:33:26,426][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:33:26,938][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:33:27,455][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:33:27,957][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:33:28,459][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:33:28,960][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:33:29,464][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:33:29,966][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:33:30,466][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:33:30,964][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:33:31,464][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:33:31,962][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:33:32,459][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:33:32,954][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:33:33,448][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:33:33,959][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:33:34,462][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:33:34,962][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:33:35,465][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:33:35,965][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:33:36,466][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:33:36,963][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:33:37,463][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:33:37,961][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:33:38,456][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:33:38,955][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:33:39,450][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:33:39,946][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:33:40,446][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:33:40,943][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:33:41,439][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:33:41,954][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:33:42,449][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:33:42,946][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:33:43,440][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:33:43,938][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:33:44,446][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:33:44,939][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:33:45,434][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:33:45,934][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:33:46,443][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:33:46,952][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:33:47,449][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:33:47,943][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:33:48,447][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:33:48,942][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:33:49,438][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:33:49,938][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:33:50,435][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:33:50,931][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:33:51,424][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:33:51,923][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:33:52,441][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:33:52,941][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:33:53,436][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:33:53,936][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:33:54,433][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:33:54,941][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:33:55,437][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:33:55,934][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:33:56,431][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:33:56,931][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:33:57,428][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10812 tokens.
[2025-11-12 23:33:58,108][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.52%, ΔTime: 00:00:32
[2025-11-12 23:33:58,897][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:33:58,898][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:33:58,900][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:33:59,907][__main__][INFO] - Iteration 95 took 51s (29.94% Gen, 68.11% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 32m 52s. Estimated total time: 42h 57m 11s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 54s, 500 more iterations: 7h 9m 31s.
[2025-11-12 23:33:59,909][__main__][INFO] - Starting iteration 95.
[2025-11-12 23:34:00,446][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:34:00,446][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:34:03,303][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:34:05,270][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:34:16,075][__main__][INFO] - Number of regex retries in iteration 95: 2
[2025-11-12 23:34:16,076][__main__][INFO] - agents played in iteration 95 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:34:16,919][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:34:16,943][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:34:16,965][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:34:16,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:34:16,987][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:34:16,988][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:34:17,667][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:34:18,127][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:34:18,643][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:34:19,148][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:34:19,657][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:34:20,158][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:34:20,658][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:34:21,161][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:34:21,663][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:34:22,165][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:34:22,676][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:34:23,181][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:34:23,684][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:34:24,184][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:34:24,682][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:34:25,183][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:34:25,685][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:34:26,184][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:34:26,683][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:34:27,181][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:34:27,681][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:34:28,180][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:34:28,680][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:34:29,184][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:34:29,684][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:34:30,183][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:34:30,688][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:34:31,189][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:34:31,690][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:34:32,189][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:34:32,690][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:34:33,190][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:34:33,690][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:34:34,186][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:34:34,686][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:34:35,182][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:34:35,678][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:34:36,176][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:34:36,674][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:34:37,177][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:34:37,674][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:34:38,178][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:34:38,680][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:34:39,181][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:34:39,681][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:34:40,182][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:34:40,683][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:34:41,184][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:34:41,687][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:34:42,186][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:34:42,684][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:34:43,182][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:34:43,683][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:34:44,181][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:34:44,681][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:34:45,184][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:34:45,680][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:34:46,179][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:34:46,677][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:34:47,178][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:34:47,682][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:34:48,179][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:34:48,676][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:34:49,176][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:34:49,674][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10812 tokens.
[2025-11-12 23:34:50,314][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-12 23:34:51,083][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:34:51,085][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:34:51,086][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:34:52,047][__main__][INFO] - Iteration 96 took 51s (30.29% Gen, 67.85% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 34m 55s. Estimated total time: 43h 0m 6s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 0s, 500 more iterations: 7h 10m 1s.
[2025-11-12 23:34:52,049][__main__][INFO] - Starting iteration 96.
[2025-11-12 23:34:52,521][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:34:52,521][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:35:03,066][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:35:08,531][__main__][INFO] - Number of regex retries in iteration 96: 1
[2025-11-12 23:35:08,532][__main__][INFO] - agents played in iteration 96 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:35:09,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:35:09,411][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:35:09,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:35:09,458][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:35:09,459][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:35:09,459][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:35:10,115][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:35:10,573][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:35:11,077][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:35:11,583][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:35:12,083][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:35:12,586][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:35:13,086][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:35:13,585][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:35:14,087][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:35:14,593][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:35:15,096][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:35:15,598][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:35:16,100][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:35:16,600][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:35:17,097][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:35:17,596][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:35:18,095][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:35:18,591][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:35:19,087][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:35:19,585][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:35:20,081][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:35:20,578][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:35:21,076][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:35:21,569][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:35:22,076][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:35:22,571][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:35:23,065][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:35:23,559][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:35:24,053][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:35:24,547][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:35:25,045][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:35:25,539][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:35:26,051][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:35:26,548][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:35:27,046][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:35:27,549][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:35:28,046][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:35:28,553][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:35:29,051][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:35:29,548][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:35:30,047][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:35:30,545][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:35:31,045][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:35:31,549][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:35:32,045][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:35:32,567][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:35:33,063][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:35:33,560][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:35:34,058][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:35:34,556][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:35:35,063][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:35:35,559][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:35:36,056][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:35:36,567][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:35:37,063][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:35:37,558][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:35:38,059][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:35:38,579][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:35:39,081][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:35:39,578][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:35:40,076][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:35:40,582][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:35:41,077][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:35:41,580][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:35:42,084][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10807 tokens.
[2025-11-12 23:35:42,771][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.29%, Current % of VRAM taken: 60.55%, Block Peak % of device VRAM: 62.45%, ΔTime: 00:00:32
[2025-11-12 23:35:43,558][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:35:43,560][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:35:43,561][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:35:44,516][__main__][INFO] - Iteration 97 took 51s (30.79% Gen, 67.37% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 53m 44s. Estimated total time: 43h 19m 47s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 39s, 500 more iterations: 7h 13m 17s.
[2025-11-12 23:35:44,518][__main__][INFO] - Starting iteration 97.
[2025-11-12 23:35:45,004][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:35:45,005][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:35:59,746][__main__][INFO] - Number of regex retries in iteration 97: 0
[2025-11-12 23:35:59,746][__main__][INFO] - agents played in iteration 97 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:36:00,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:36:00,765][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:36:00,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:36:00,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:36:00,813][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:36:00,814][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:36:01,497][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:36:01,956][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:36:02,456][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:36:02,975][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:36:03,470][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:36:03,967][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:36:04,472][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:36:04,968][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:36:05,474][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:36:05,978][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:36:06,481][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:36:06,986][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:36:07,486][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:36:07,983][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:36:08,479][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:36:08,976][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:36:09,479][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:36:09,974][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:36:10,470][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:36:10,970][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:36:11,463][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:36:11,958][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:36:12,461][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:36:12,957][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:36:13,466][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:36:13,962][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:36:14,459][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:36:14,957][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:36:15,453][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:36:15,950][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:36:16,448][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:36:16,948][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:36:17,468][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:36:17,986][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:36:18,490][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:36:18,993][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:36:19,491][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:36:19,993][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:36:20,494][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:36:20,997][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:36:21,498][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:36:21,995][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:36:22,489][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:36:22,985][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:36:23,479][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:36:23,985][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:36:24,479][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:36:24,977][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:36:25,483][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:36:25,982][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:36:26,480][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:36:26,981][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:36:27,479][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:36:27,985][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:36:28,483][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:36:28,980][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:36:29,477][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:36:29,973][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:36:30,479][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:36:30,976][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:36:31,473][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:36:31,972][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:36:32,469][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:36:32,965][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:36:33,462][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10665 tokens.
[2025-11-12 23:36:34,200][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.38%, ΔTime: 00:00:32
[2025-11-12 23:36:34,995][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:36:34,996][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:36:34,998][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:36:35,968][__main__][INFO] - Iteration 98 took 50s (28.92% Gen, 69.17% Train). Generation: 14s, Training: 35s. Estimated remaining time: 41h 1m 19s. Estimated total time: 42h 28m 13s. Time estimates for 10 more iterations: 8m 29s, 100 more iterations: 1h 24m 56s, 500 more iterations: 7h 4m 42s.
[2025-11-12 23:36:35,970][__main__][INFO] - Starting iteration 98.
[2025-11-12 23:36:36,534][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:36:36,535][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:36:40,853][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 30 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:36:52,736][__main__][INFO] - Number of regex retries in iteration 98: 1
[2025-11-12 23:36:52,737][__main__][INFO] - agents played in iteration 98 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:36:53,697][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:36:53,724][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:36:53,752][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:36:53,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:36:53,776][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:36:53,777][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:36:54,462][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:36:54,927][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:36:55,432][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:36:55,931][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:36:56,436][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:36:56,936][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:36:57,438][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:36:57,937][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:36:58,438][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:36:58,940][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:36:59,442][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:36:59,941][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:37:00,439][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:37:00,952][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:37:01,455][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:37:01,955][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:37:02,455][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:37:02,963][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:37:03,460][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:37:03,960][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:37:04,456][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:37:04,952][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:37:05,450][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:37:05,943][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:37:06,438][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:37:06,935][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:37:07,432][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:37:07,929][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:37:08,421][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:37:08,921][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:37:09,419][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:37:09,914][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:37:10,409][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:37:10,905][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:37:11,397][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:37:11,901][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:37:12,397][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:37:12,891][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:37:13,386][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:37:13,881][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:37:14,378][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:37:14,872][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:37:15,369][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:37:15,881][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:37:16,377][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:37:16,874][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:37:17,374][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:37:17,870][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:37:18,379][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:37:18,877][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:37:19,374][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:37:19,894][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:37:20,392][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:37:20,889][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:37:21,385][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:37:21,881][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:37:22,387][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:37:22,880][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:37:23,377][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:37:23,876][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:37:24,373][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:37:24,869][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:37:25,374][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:37:25,874][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:37:26,374][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10748 tokens.
[2025-11-12 23:37:27,051][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-12 23:37:27,834][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:37:27,835][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:37:27,837][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:37:28,780][__main__][INFO] - Iteration 99 took 52s (31.01% Gen, 67.18% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 4m 33s. Estimated total time: 43h 32m 20s. Time estimates for 10 more iterations: 8m 42s, 100 more iterations: 1h 27m 4s, 500 more iterations: 7h 15m 23s.
[2025-11-12 23:37:28,783][__main__][INFO] - Starting iteration 99.
[2025-11-12 23:37:29,311][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:37:29,311][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:37:45,105][__main__][INFO] - Number of regex retries in iteration 99: 0
[2025-11-12 23:37:45,106][__main__][INFO] - agents played in iteration 99 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:37:45,978][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:37:46,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:37:46,027][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:37:46,050][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:37:46,050][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:37:46,051][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:37:46,686][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:37:47,145][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:37:47,648][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:37:48,150][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:37:48,658][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:37:49,157][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:37:49,656][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:37:50,159][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:37:50,661][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:37:51,166][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:37:51,665][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:37:52,164][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:37:52,665][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:37:53,166][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:37:53,666][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:37:54,171][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:37:54,672][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:37:55,172][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:37:55,674][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:37:56,177][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:37:56,693][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:37:57,192][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:37:57,691][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:37:58,189][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:37:58,689][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:37:59,194][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:37:59,693][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:38:00,195][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:38:00,694][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:38:01,192][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:38:01,692][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:38:02,196][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:38:02,694][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:38:03,203][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:38:03,701][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:38:04,201][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:38:04,695][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:38:05,191][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:38:05,696][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:38:06,195][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:38:06,694][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:38:07,199][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:38:07,696][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:38:08,194][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:38:08,690][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:38:09,186][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:38:09,687][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:38:10,184][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:38:10,679][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:38:11,180][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:38:11,677][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:38:12,175][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:38:12,673][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:38:13,170][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:38:13,679][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:38:14,176][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:38:14,674][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:38:15,186][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:38:15,680][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:38:16,181][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:38:16,689][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:38:17,186][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:38:17,700][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:38:18,198][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:38:18,700][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10864 tokens.
[2025-11-12 23:38:19,405][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-12 23:38:20,175][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:38:20,176][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:38:20,179][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:38:21,046][__main__][INFO] - Iteration 100 took 51s (30.53% Gen, 67.79% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 38m 9s. Estimated total time: 43h 6m 49s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 13s, 500 more iterations: 7h 11m 8s.
[2025-11-12 23:38:21,048][__main__][INFO] - Starting iteration 100.
[2025-11-12 23:38:21,542][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-12 23:38:21,542][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:38:28,598][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:38:36,912][__main__][INFO] - Number of regex retries in iteration 100: 1
[2025-11-12 23:38:36,913][__main__][INFO] - agents played in iteration 100 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:38:37,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:38:37,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:38:37,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:38:37,842][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:38:37,843][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:38:37,844][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:38:38,460][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:38:38,917][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:38:39,418][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:38:39,914][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:38:40,417][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:38:40,919][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:38:41,419][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:38:41,916][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:38:42,417][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:38:42,927][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:38:43,427][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:38:43,931][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:38:44,429][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:38:44,926][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:38:45,426][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:38:45,923][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:38:46,420][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:38:46,920][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:38:47,416][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:38:47,918][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:38:48,417][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:38:48,914][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:38:49,421][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:38:49,917][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:38:50,411][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:38:50,908][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:38:51,402][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:38:51,908][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:38:52,405][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:38:52,902][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:38:53,403][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:38:53,900][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:38:54,397][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:38:54,892][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:38:55,390][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:38:55,909][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:38:56,404][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:38:56,901][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:38:57,402][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:38:57,897][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:38:58,394][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:38:58,892][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:38:59,390][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:38:59,901][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:39:00,397][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:39:00,895][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:39:01,395][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:39:01,891][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:39:02,393][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:39:02,886][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:39:03,382][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:39:03,880][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:39:04,377][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:39:04,874][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:39:05,371][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:39:05,865][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:39:06,361][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:39:06,857][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:39:07,358][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:39:07,860][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:39:08,360][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:39:08,861][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:39:09,362][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:39:09,864][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:39:10,367][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10760 tokens.
[2025-11-12 23:39:11,061][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-12 23:39:11,810][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:39:11,811][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:39:11,813][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:39:13,673][__main__][INFO] - Iteration 101 took 52s (29.48% Gen, 66.95% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 57m 5s. Estimated total time: 43h 26m 37s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 53s, 500 more iterations: 7h 14m 26s.
[2025-11-12 23:39:13,676][__main__][INFO] - Starting iteration 101.
[2025-11-12 23:39:14,177][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:39:14,178][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:39:17,071][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:39:29,456][__main__][INFO] - Number of regex retries in iteration 101: 1
[2025-11-12 23:39:29,457][__main__][INFO] - agents played in iteration 101 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:39:30,274][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:39:30,301][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:39:30,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:39:30,351][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:39:30,352][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:39:30,353][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:39:30,965][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:39:31,428][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:39:31,931][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:39:32,433][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:39:32,932][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:39:33,425][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:39:33,926][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:39:34,421][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:39:34,918][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:39:35,416][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:39:35,914][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:39:36,413][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:39:36,914][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:39:37,417][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:39:37,930][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:39:38,427][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:39:38,924][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:39:39,425][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:39:39,922][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:39:40,442][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:39:40,939][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:39:41,435][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:39:41,937][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:39:42,436][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:39:42,934][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:39:43,436][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:39:43,935][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:39:44,436][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:39:44,933][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:39:45,430][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:39:45,932][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:39:46,428][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:39:46,931][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:39:47,425][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:39:47,922][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:39:48,428][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:39:48,925][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:39:49,422][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:39:49,927][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:39:50,424][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:39:50,921][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:39:51,417][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:39:51,914][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:39:52,420][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:39:52,917][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:39:53,414][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:39:53,929][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:39:54,424][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:39:54,919][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:39:55,417][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:39:55,916][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:39:56,429][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:39:56,926][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:39:57,423][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:39:57,931][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:39:58,433][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:39:58,936][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:39:59,436][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:39:59,936][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:40:00,439][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:40:00,937][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:40:01,438][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:40:01,941][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:40:02,442][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:40:02,943][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10807 tokens.
[2025-11-12 23:40:03,636][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.42%, ΔTime: 00:00:32
[2025-11-12 23:40:04,408][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:40:04,410][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:40:04,411][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:40:05,587][__main__][INFO] - Iteration 102 took 51s (29.72% Gen, 67.99% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 20m 7s. Estimated total time: 42h 50m 31s. Time estimates for 10 more iterations: 8m 34s, 100 more iterations: 1h 25m 41s, 500 more iterations: 7h 8m 25s.
[2025-11-12 23:40:05,589][__main__][INFO] - Starting iteration 102.
[2025-11-12 23:40:06,083][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:40:06,084][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:40:09,057][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 5 hats, 5 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:40:11,943][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:40:14,331][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:40:18,439][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:40:22,782][__main__][INFO] - Number of regex retries in iteration 102: 4
[2025-11-12 23:40:22,782][__main__][INFO] - agents played in iteration 102 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:40:23,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:40:23,648][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:40:23,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:40:23,698][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:40:23,699][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:40:23,700][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:40:24,323][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:40:24,775][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:40:25,284][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:40:25,786][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:40:26,283][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:40:26,780][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:40:27,279][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:40:27,780][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:40:28,280][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:40:28,782][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:40:29,286][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:40:29,801][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:40:30,311][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:40:30,833][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:40:31,336][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:40:31,831][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:40:32,334][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:40:32,832][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:40:33,327][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:40:33,828][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:40:34,328][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:40:34,846][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:40:35,342][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:40:35,839][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:40:36,336][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:40:36,839][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:40:37,357][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:40:37,854][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:40:38,352][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:40:38,853][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:40:39,353][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:40:39,854][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:40:40,350][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:40:40,848][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:40:41,352][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:40:41,849][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:40:42,346][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:40:42,844][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:40:43,339][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:40:43,837][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:40:44,337][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:40:44,835][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:40:45,343][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:40:45,840][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:40:46,337][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:40:46,836][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:40:47,339][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:40:47,838][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:40:48,336][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:40:48,833][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:40:49,345][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:40:49,844][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:40:50,344][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:40:50,845][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:40:51,342][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:40:51,843][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:40:52,341][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:40:52,840][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:40:53,340][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:40:53,841][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:40:54,340][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:40:54,843][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:40:55,342][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:40:55,851][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:40:56,352][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10750 tokens.
[2025-11-12 23:40:57,022][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.38%, ΔTime: 00:00:32
[2025-11-12 23:40:57,745][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:40:57,747][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:40:57,748][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:40:58,651][__main__][INFO] - Iteration 103 took 52s (31.76% Gen, 66.52% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 17m 7s. Estimated total time: 43h 48m 24s. Time estimates for 10 more iterations: 8m 45s, 100 more iterations: 1h 27m 36s, 500 more iterations: 7h 18m 4s.
[2025-11-12 23:40:58,653][__main__][INFO] - Starting iteration 103.
[2025-11-12 23:40:59,125][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:40:59,125][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:41:01,921][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:41:02,000][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:41:15,230][__main__][INFO] - Number of regex retries in iteration 103: 2
[2025-11-12 23:41:15,231][__main__][INFO] - agents played in iteration 103 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:41:16,099][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:41:16,127][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:41:16,154][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:41:16,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:41:16,177][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:41:16,179][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:41:16,802][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:41:17,256][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:41:17,762][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:41:18,270][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:41:18,772][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:41:19,275][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:41:19,775][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:41:20,275][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:41:20,778][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:41:21,283][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:41:21,784][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:41:22,288][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:41:22,787][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:41:23,288][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:41:23,786][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:41:24,287][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:41:24,787][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:41:25,284][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:41:25,782][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:41:26,279][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:41:26,777][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:41:27,282][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:41:27,782][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:41:28,279][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:41:28,778][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:41:29,277][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:41:29,774][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:41:30,268][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:41:30,767][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:41:31,266][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:41:31,764][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:41:32,261][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:41:32,761][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:41:33,256][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:41:33,751][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:41:34,247][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:41:34,741][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:41:35,251][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:41:35,749][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:41:36,245][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:41:36,742][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:41:37,241][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:41:37,745][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:41:38,245][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:41:38,746][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:41:39,244][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:41:39,741][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:41:40,243][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:41:40,745][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:41:41,267][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:41:41,776][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:41:42,278][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:41:42,782][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:41:43,282][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:41:43,786][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:41:44,292][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:41:44,792][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:41:45,292][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:41:45,793][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:41:46,295][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:41:46,798][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:41:47,301][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:41:47,801][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:41:48,300][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:41:48,802][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10775 tokens.
[2025-11-12 23:41:49,478][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.38%, ΔTime: 00:00:32
[2025-11-12 23:41:50,209][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:41:50,211][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:41:50,212][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:41:51,154][__main__][INFO] - Iteration 104 took 52s (30.95% Gen, 67.23% Train). Generation: 16s, Training: 34s. Estimated remaining time: 41h 49m 21s. Estimated total time: 43h 21m 30s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 43s, 500 more iterations: 7h 13m 35s.
[2025-11-12 23:41:51,157][__main__][INFO] - Starting iteration 104.
[2025-11-12 23:41:51,629][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:41:51,629][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:42:01,569][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:42:06,868][__main__][INFO] - Number of regex retries in iteration 104: 1
[2025-11-12 23:42:06,869][__main__][INFO] - agents played in iteration 104 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:42:07,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:42:07,688][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:42:07,713][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:42:07,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:42:07,735][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:42:07,737][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:42:08,371][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:42:08,829][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:42:09,332][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:42:09,842][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:42:10,342][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:42:10,857][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:42:11,355][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:42:11,854][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:42:12,356][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:42:12,856][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:42:13,367][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:42:13,868][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:42:14,366][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:42:14,871][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:42:15,368][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:42:15,866][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:42:16,364][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:42:16,863][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:42:17,363][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:42:17,857][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:42:18,355][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:42:18,867][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:42:19,365][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:42:19,862][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:42:20,360][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:42:20,858][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:42:21,365][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:42:21,863][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:42:22,361][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:42:22,866][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:42:23,361][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:42:23,862][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:42:24,361][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:42:24,864][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:42:25,371][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:42:25,870][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:42:26,371][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:42:26,876][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:42:27,377][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:42:27,877][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:42:28,378][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:42:28,876][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:42:29,376][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:42:29,879][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:42:30,381][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:42:30,884][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:42:31,384][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:42:31,889][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:42:32,391][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:42:32,891][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:42:33,396][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:42:33,893][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:42:34,394][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:42:34,895][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:42:35,394][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:42:35,896][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:42:36,396][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:42:36,898][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:42:37,404][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:42:37,905][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:42:38,403][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:42:38,905][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:42:39,407][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:42:39,906][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:42:40,406][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10782 tokens.
[2025-11-12 23:42:41,086][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-12 23:42:41,831][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:42:41,832][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:42:41,834][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:42:42,765][__main__][INFO] - Iteration 105 took 51s (29.80% Gen, 68.38% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 3m 48s. Estimated total time: 42h 36m 49s. Time estimates for 10 more iterations: 8m 31s, 100 more iterations: 1h 25m 13s, 500 more iterations: 7h 6m 8s.
[2025-11-12 23:42:42,767][__main__][INFO] - Starting iteration 105.
[2025-11-12 23:42:43,254][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:42:43,255][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:42:52,525][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:42:59,017][__main__][INFO] - Number of regex retries in iteration 105: 1
[2025-11-12 23:42:59,018][__main__][INFO] - agents played in iteration 105 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:42:59,819][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:42:59,841][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:42:59,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:42:59,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:42:59,886][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:42:59,887][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:43:00,515][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:43:00,973][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:43:01,480][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:43:01,985][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:43:02,489][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:43:02,990][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:43:03,490][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:43:03,991][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:43:04,490][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:43:04,993][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:43:05,497][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:43:06,002][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:43:06,499][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:43:06,999][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:43:07,498][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:43:07,999][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:43:08,499][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:43:08,998][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:43:09,498][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:43:10,000][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:43:10,501][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:43:11,000][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:43:11,499][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:43:11,998][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:43:12,498][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:43:13,000][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:43:13,498][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:43:13,998][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:43:14,495][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:43:14,997][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:43:15,497][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:43:15,995][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:43:16,493][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:43:16,996][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:43:17,495][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:43:17,996][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:43:18,494][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:43:18,994][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:43:19,495][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:43:19,994][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:43:20,493][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:43:20,999][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:43:21,504][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:43:22,011][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:43:22,521][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:43:23,025][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:43:23,529][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:43:24,049][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:43:24,554][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:43:25,057][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:43:25,559][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:43:26,078][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:43:26,580][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:43:27,090][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:43:27,603][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:43:28,106][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:43:28,611][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:43:29,111][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:43:29,613][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:43:30,115][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:43:30,616][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:43:31,116][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:43:31,613][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:43:32,112][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:43:32,615][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10837 tokens.
[2025-11-12 23:43:33,261][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.51%, ΔTime: 00:00:32
[2025-11-12 23:43:34,035][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:43:34,037][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:43:34,039][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:43:35,008][__main__][INFO] - Iteration 106 took 51s (30.46% Gen, 67.67% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 33m 50s. Estimated total time: 43h 7m 43s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 15s, 500 more iterations: 7h 11m 17s.
[2025-11-12 23:43:35,011][__main__][INFO] - Starting iteration 106.
[2025-11-12 23:43:35,830][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:43:35,831][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:43:52,403][__main__][INFO] - Number of regex retries in iteration 106: 0
[2025-11-12 23:43:52,403][__main__][INFO] - agents played in iteration 106 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:43:53,258][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:43:53,287][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:43:53,324][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:43:53,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:43:53,349][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:43:53,350][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:43:53,976][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:43:54,431][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:43:54,941][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:43:55,442][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:43:55,995][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:43:56,494][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:43:56,995][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:43:57,508][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:43:58,012][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:43:58,514][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:43:59,012][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:43:59,511][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:44:00,011][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:44:00,509][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:44:01,006][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:44:01,508][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:44:02,005][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:44:02,503][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:44:02,997][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:44:03,494][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:44:04,004][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:44:04,502][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:44:04,998][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:44:05,496][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:44:05,992][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:44:06,499][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:44:06,997][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:44:07,495][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:44:07,995][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:44:08,492][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:44:08,990][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:44:09,486][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:44:09,982][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:44:10,499][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:44:10,995][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:44:11,493][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:44:11,991][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:44:12,495][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:44:13,003][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:44:13,502][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:44:14,002][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:44:14,501][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:44:15,004][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:44:15,512][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:44:16,009][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:44:16,509][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:44:17,014][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:44:17,515][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:44:18,012][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:44:18,506][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:44:19,003][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:44:19,510][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:44:20,011][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:44:20,513][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:44:21,018][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:44:21,516][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:44:22,022][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:44:22,522][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:44:23,023][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:44:23,522][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:44:24,023][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:44:24,523][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:44:25,020][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:44:25,519][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:44:26,021][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10817 tokens.
[2025-11-12 23:44:26,676][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-12 23:44:27,452][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:44:27,454][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:44:27,456][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:44:28,431][__main__][INFO] - Iteration 107 took 52s (31.51% Gen, 66.64% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 15m 16s. Estimated total time: 43h 50m 3s. Time estimates for 10 more iterations: 8m 46s, 100 more iterations: 1h 27m 40s, 500 more iterations: 7h 18m 20s.
[2025-11-12 23:44:28,433][__main__][INFO] - Starting iteration 107.
[2025-11-12 23:44:28,914][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:44:28,915][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:44:31,232][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:44:31,233][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:44:42,784][__main__][INFO] - Number of regex retries in iteration 107: 2
[2025-11-12 23:44:42,784][__main__][INFO] - agents played in iteration 107 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:44:43,607][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:44:43,632][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:44:43,656][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:44:43,678][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:44:43,679][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:44:43,680][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:44:44,317][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:44:44,775][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:44:45,280][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:44:45,798][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:44:46,298][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:44:46,802][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:44:47,306][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:44:47,804][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:44:48,311][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:44:48,809][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:44:49,308][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:44:49,810][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:44:50,306][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:44:50,804][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:44:51,300][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:44:51,797][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:44:52,302][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:44:52,802][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:44:53,297][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:44:53,798][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:44:54,298][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:44:54,799][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:44:55,300][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:44:55,800][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:44:56,309][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:44:56,807][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:44:57,305][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:44:57,805][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:44:58,303][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:44:58,806][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:44:59,304][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:44:59,802][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:45:00,308][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:45:00,810][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:45:01,312][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:45:01,813][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:45:02,317][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:45:02,821][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:45:03,322][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:45:03,821][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:45:04,325][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:45:04,826][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:45:05,330][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:45:05,830][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:45:06,330][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:45:06,830][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:45:07,329][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:45:07,828][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:45:08,328][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:45:08,826][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:45:09,324][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:45:09,821][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:45:10,325][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:45:10,831][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:45:11,333][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:45:11,833][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:45:12,337][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:45:12,837][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:45:13,344][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:45:13,848][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:45:14,349][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:45:14,852][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:45:15,351][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:45:15,852][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:45:16,356][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10755 tokens.
[2025-11-12 23:45:16,997][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-12 23:45:17,758][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:45:17,760][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:45:17,762][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:45:18,689][__main__][INFO] - Iteration 108 took 49s (27.86% Gen, 70.27% Train). Generation: 13s, Training: 34s. Estimated remaining time: 39h 53m 9s. Estimated total time: 41h 28m 46s. Time estimates for 10 more iterations: 8m 17s, 100 more iterations: 1h 22m 57s, 500 more iterations: 6h 54m 47s.
[2025-11-12 23:45:18,691][__main__][INFO] - Starting iteration 108.
[2025-11-12 23:45:19,159][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:45:19,159][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:45:33,624][__main__][INFO] - Number of regex retries in iteration 108: 0
[2025-11-12 23:45:33,625][__main__][INFO] - agents played in iteration 108 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:45:34,434][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:45:34,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:45:34,484][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:45:34,506][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:45:34,506][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:45:34,507][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:45:35,117][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:45:35,775][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:45:36,277][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:45:36,777][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:45:37,283][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:45:37,783][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:45:38,295][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:45:38,793][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:45:39,293][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:45:39,799][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:45:40,300][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:45:40,806][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:45:41,302][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:45:41,800][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:45:42,300][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:45:42,797][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:45:43,294][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:45:43,787][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:45:44,286][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:45:44,797][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:45:45,297][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:45:45,794][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:45:46,297][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:45:46,794][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:45:47,294][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:45:47,792][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:45:48,290][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:45:48,803][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:45:49,300][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:45:49,796][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:45:50,297][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:45:50,796][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:45:51,307][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:45:51,814][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:45:52,320][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:45:52,823][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:45:53,325][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:45:53,827][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:45:54,328][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:45:54,828][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:45:55,329][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:45:55,829][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:45:56,329][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:45:56,827][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:45:57,324][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:45:57,826][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:45:58,321][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:45:58,823][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:45:59,324][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:45:59,821][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:46:00,317][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:46:00,815][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:46:01,313][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:46:01,819][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:46:02,319][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:46:02,820][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:46:03,324][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:46:03,823][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:46:04,324][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:46:04,823][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:46:05,324][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:46:05,830][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:46:06,329][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:46:06,827][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:46:07,330][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10795 tokens.
[2025-11-12 23:46:07,979][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-12 23:46:08,743][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:46:08,744][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:46:08,748][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:46:09,670][__main__][INFO] - Iteration 109 took 50s (28.64% Gen, 69.53% Train). Generation: 14s, Training: 35s. Estimated remaining time: 40h 29m 8s. Estimated total time: 42h 5m 36s. Time estimates for 10 more iterations: 8m 25s, 100 more iterations: 1h 24m 11s, 500 more iterations: 7h 0m 56s.
[2025-11-12 23:46:09,672][__main__][INFO] - Starting iteration 109.
[2025-11-12 23:46:10,179][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:46:10,179][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:46:13,315][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:46:26,777][__main__][INFO] - Number of regex retries in iteration 109: 1
[2025-11-12 23:46:26,777][__main__][INFO] - agents played in iteration 109 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:46:27,611][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:46:27,639][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:46:27,665][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:46:27,691][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:46:27,692][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:46:27,692][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:46:28,342][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:46:28,799][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:46:29,318][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:46:29,817][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:46:30,319][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:46:30,824][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:46:31,327][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:46:31,834][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:46:32,335][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:46:32,837][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:46:33,342][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:46:33,839][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:46:34,338][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:46:34,837][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:46:35,339][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:46:35,835][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:46:36,332][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:46:36,831][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:46:37,338][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:46:37,836][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:46:38,337][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:46:38,837][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:46:39,334][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:46:39,838][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:46:40,336][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:46:40,832][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:46:41,333][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:46:41,829][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:46:42,326][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:46:42,828][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:46:43,327][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:46:43,837][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:46:44,338][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:46:44,840][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:46:45,344][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:46:45,841][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:46:46,344][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:46:46,846][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:46:47,348][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:46:47,851][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:46:48,351][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:46:48,852][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:46:49,354][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:46:49,849][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:46:50,350][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:46:50,849][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:46:51,347][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:46:51,846][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:46:52,342][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:46:52,842][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:46:53,345][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:46:53,842][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:46:54,338][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:46:54,837][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:46:55,335][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:46:55,838][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:46:56,339][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:46:56,842][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:46:57,338][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:46:57,837][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:46:58,341][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:46:58,843][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:46:59,343][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:46:59,844][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:47:00,344][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10767 tokens.
[2025-11-12 23:47:01,035][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-12 23:47:01,782][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:47:01,784][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:47:01,787][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:47:02,987][__main__][INFO] - Iteration 110 took 52s (31.43% Gen, 66.29% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 23m 5s. Estimated total time: 44h 0m 26s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 0s, 500 more iterations: 7h 20m 4s.
[2025-11-12 23:47:02,989][__main__][INFO] - Starting iteration 110.
[2025-11-12 23:47:03,461][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-12 23:47:03,463][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:47:07,141][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 30 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:47:18,343][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:47:19,161][__main__][INFO] - Number of regex retries in iteration 110: 2
[2025-11-12 23:47:19,162][__main__][INFO] - agents played in iteration 110 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:47:20,013][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:47:20,043][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:47:20,069][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:47:20,092][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:47:20,092][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:47:20,093][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:47:20,726][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:47:21,183][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:47:21,685][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:47:22,191][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:47:22,691][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:47:23,191][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:47:23,688][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:47:24,186][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:47:24,687][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:47:25,183][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:47:25,685][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:47:26,187][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:47:26,686][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:47:27,184][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:47:27,681][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:47:28,178][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:47:28,677][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:47:29,172][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:47:29,670][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:47:30,166][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:47:30,664][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:47:31,161][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:47:31,659][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:47:32,158][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:47:32,656][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:47:33,150][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:47:33,649][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:47:34,147][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:47:34,645][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:47:35,145][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:47:35,644][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:47:36,140][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:47:36,640][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:47:37,139][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:47:37,646][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:47:38,146][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:47:38,648][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:47:39,147][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:47:39,645][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:47:40,147][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:47:40,644][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:47:41,141][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:47:41,649][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:47:42,147][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:47:42,645][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:47:43,143][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:47:43,641][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:47:44,149][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:47:44,648][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:47:45,147][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:47:45,644][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:47:46,141][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:47:46,643][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:47:47,146][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:47:47,644][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:47:48,141][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:47:48,638][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:47:49,140][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:47:49,640][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:47:50,140][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:47:50,651][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:47:51,151][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:47:51,650][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:47:52,153][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:47:52,649][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10758 tokens.
[2025-11-12 23:47:53,373][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-12 23:47:54,108][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:47:54,112][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:47:54,114][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:47:55,825][__main__][INFO] - Iteration 111 took 52s (29.98% Gen, 66.75% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 59m 58s. Estimated total time: 43h 38m 12s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 16s, 500 more iterations: 7h 16m 22s.
[2025-11-12 23:47:55,827][__main__][INFO] - Starting iteration 111.
[2025-11-12 23:47:56,319][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:47:56,319][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:48:05,943][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:48:11,906][__main__][INFO] - Number of regex retries in iteration 111: 1
[2025-11-12 23:48:11,906][__main__][INFO] - agents played in iteration 111 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:48:12,711][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:48:12,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:48:12,769][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:48:12,796][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:48:12,796][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:48:12,797][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:48:13,422][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:48:13,878][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:48:14,391][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:48:14,891][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:48:15,389][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:48:15,888][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:48:16,387][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:48:16,889][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:48:17,388][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:48:17,886][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:48:18,385][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:48:18,882][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:48:19,379][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:48:19,876][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:48:20,374][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:48:20,876][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:48:21,375][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:48:21,874][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:48:22,371][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:48:22,867][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:48:23,366][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:48:23,863][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:48:24,361][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:48:24,858][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:48:25,357][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:48:25,858][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:48:26,355][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:48:26,856][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:48:27,358][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:48:27,857][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:48:28,357][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:48:28,854][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:48:29,355][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:48:29,859][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:48:30,359][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:48:30,860][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:48:31,363][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:48:31,863][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:48:32,368][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:48:32,867][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:48:33,366][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:48:33,867][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:48:34,365][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:48:34,863][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:48:35,361][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:48:35,860][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:48:36,367][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:48:36,865][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:48:37,368][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:48:37,862][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:48:38,359][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:48:38,861][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:48:39,359][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:48:39,857][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:48:40,358][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:48:40,856][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:48:41,356][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:48:41,857][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:48:42,361][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:48:42,871][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:48:43,375][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:48:43,876][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:48:44,375][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:48:44,876][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:48:45,384][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10828 tokens.
[2025-11-12 23:48:46,084][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-12 23:48:46,825][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:48:46,826][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:48:46,828][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:48:47,744][__main__][INFO] - Iteration 112 took 51s (30.31% Gen, 67.91% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 12m 13s. Estimated total time: 42h 51m 19s. Time estimates for 10 more iterations: 8m 34s, 100 more iterations: 1h 25m 42s, 500 more iterations: 7h 8m 33s.
[2025-11-12 23:48:47,746][__main__][INFO] - Starting iteration 112.
[2025-11-12 23:48:48,233][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:48:48,234][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:49:04,827][__main__][INFO] - Number of regex retries in iteration 112: 0
[2025-11-12 23:49:04,828][__main__][INFO] - agents played in iteration 112 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:49:05,639][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:49:05,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:49:05,697][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:49:05,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:49:05,721][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:49:05,722][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:49:06,351][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:49:06,809][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:49:07,317][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:49:07,820][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:49:08,320][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:49:08,818][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:49:09,316][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:49:09,815][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:49:10,316][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:49:10,814][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:49:11,310][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:49:11,805][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:49:12,301][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:49:12,798][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:49:13,295][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:49:13,792][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:49:14,296][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:49:14,793][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:49:15,287][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:49:15,781][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:49:16,278][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:49:16,779][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:49:17,277][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:49:17,777][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:49:18,285][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:49:18,780][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:49:19,279][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:49:19,781][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:49:20,279][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:49:20,785][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:49:21,285][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:49:21,781][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:49:22,288][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:49:22,785][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:49:23,282][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:49:23,782][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:49:24,278][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:49:24,782][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:49:25,278][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:49:25,776][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:49:26,271][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:49:26,770][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:49:27,276][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:49:27,775][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:49:28,275][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:49:28,785][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:49:29,284][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:49:29,783][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:49:30,280][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:49:30,779][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:49:31,283][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:49:31,780][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:49:32,279][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:49:32,779][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:49:33,279][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:49:33,780][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:49:34,280][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:49:34,786][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:49:35,295][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:49:35,794][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:49:36,300][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:49:36,803][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:49:37,306][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:49:37,807][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:49:38,307][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10757 tokens.
[2025-11-12 23:49:38,977][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.29%, Current % of VRAM taken: 60.55%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-12 23:49:39,722][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:49:39,724][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:49:39,726][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:49:40,693][__main__][INFO] - Iteration 113 took 52s (31.63% Gen, 66.52% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 3m 2s. Estimated total time: 43h 43m 1s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 26s, 500 more iterations: 7h 17m 10s.
[2025-11-12 23:49:40,695][__main__][INFO] - Starting iteration 113.
[2025-11-12 23:49:41,177][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:49:41,178][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:49:54,296][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:49:57,926][__main__][INFO] - Number of regex retries in iteration 113: 1
[2025-11-12 23:49:57,927][__main__][INFO] - agents played in iteration 113 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:49:58,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:49:58,841][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:49:58,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:49:58,890][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:49:58,891][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:49:58,893][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:49:59,524][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:49:59,981][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:50:00,482][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:50:00,980][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:50:01,487][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:50:01,986][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:50:02,484][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:50:02,984][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:50:03,483][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:50:03,980][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:50:04,480][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:50:04,975][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:50:05,495][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:50:05,992][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:50:06,489][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:50:06,987][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:50:07,482][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:50:07,980][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:50:08,478][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:50:08,976][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:50:09,497][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:50:09,997][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:50:10,497][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:50:11,013][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:50:11,513][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:50:12,014][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:50:12,508][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:50:13,010][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:50:13,518][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:50:14,017][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:50:14,519][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:50:15,021][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:50:15,518][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:50:16,018][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:50:16,517][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:50:17,015][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:50:17,521][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:50:18,019][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:50:18,518][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:50:19,015][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:50:19,515][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:50:20,016][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:50:20,513][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:50:21,010][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:50:21,516][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:50:22,011][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:50:22,509][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:50:23,005][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:50:23,503][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:50:24,015][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:50:24,513][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:50:25,009][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:50:25,507][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:50:26,006][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:50:26,513][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:50:27,015][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:50:27,514][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:50:28,022][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:50:28,523][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:50:29,020][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:50:29,524][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:50:30,023][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:50:30,525][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:50:31,025][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:50:31,519][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10808 tokens.
[2025-11-12 23:50:32,191][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-12 23:50:32,961][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:50:32,964][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:50:32,965][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:50:33,879][__main__][INFO] - Iteration 114 took 52s (31.78% Gen, 66.48% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 14m 15s. Estimated total time: 43h 55m 7s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 50s, 500 more iterations: 7h 19m 11s.
[2025-11-12 23:50:33,881][__main__][INFO] - Starting iteration 114.
[2025-11-12 23:50:34,371][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:50:34,372][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:50:49,831][__main__][INFO] - Number of regex retries in iteration 114: 0
[2025-11-12 23:50:49,831][__main__][INFO] - agents played in iteration 114 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:50:50,678][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:50:50,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:50:50,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:50:50,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:50:50,755][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:50:50,755][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:50:51,370][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:50:51,827][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:50:52,331][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:50:52,830][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:50:53,330][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:50:53,828][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:50:54,327][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:50:54,825][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:50:55,325][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:50:55,948][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:50:56,446][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:50:56,947][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:50:57,445][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:50:57,943][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:50:58,441][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:50:58,940][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:50:59,446][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:50:59,948][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:51:00,448][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:51:00,952][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:51:01,459][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:51:01,957][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:51:02,458][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:51:02,958][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:51:03,460][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:51:03,960][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:51:04,458][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:51:04,958][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:51:05,457][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:51:05,956][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:51:06,451][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:51:06,950][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:51:07,449][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:51:07,949][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:51:08,447][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:51:08,946][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:51:09,446][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:51:09,948][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:51:10,447][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:51:10,948][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:51:11,446][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:51:11,944][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:51:12,445][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:51:12,940][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:51:13,440][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:51:13,946][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:51:14,445][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:51:14,947][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:51:15,447][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:51:15,944][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:51:16,443][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:51:16,941][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:51:17,440][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:51:17,938][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:51:18,441][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:51:18,947][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:51:19,449][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:51:19,951][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:51:20,454][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:51:20,955][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:51:21,460][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:51:21,959][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:51:22,461][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:51:22,963][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:51:23,465][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10793 tokens.
[2025-11-12 23:51:24,151][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-12 23:51:24,914][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:51:24,917][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:51:24,918][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:51:25,810][__main__][INFO] - Iteration 115 took 51s (30.05% Gen, 68.21% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 10m 14s. Estimated total time: 42h 51m 58s. Time estimates for 10 more iterations: 8m 34s, 100 more iterations: 1h 25m 43s, 500 more iterations: 7h 8m 39s.
[2025-11-12 23:51:25,813][__main__][INFO] - Starting iteration 115.
[2025-11-12 23:51:26,281][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:51:26,281][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:51:29,052][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:51:41,825][__main__][INFO] - Number of regex retries in iteration 115: 1
[2025-11-12 23:51:41,826][__main__][INFO] - agents played in iteration 115 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:51:42,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:51:42,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:51:42,688][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:51:42,709][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:51:42,710][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:51:42,711][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:51:43,319][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:51:43,778][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:51:44,278][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:51:44,780][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:51:45,281][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:51:45,779][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:51:46,278][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:51:46,776][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:51:47,276][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:51:47,776][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:51:48,278][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:51:48,776][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:51:49,276][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:51:49,776][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:51:50,275][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:51:50,774][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:51:51,273][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:51:51,776][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:51:52,278][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:51:52,781][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:51:53,282][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:51:53,783][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:51:54,283][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:51:54,783][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:51:55,283][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:51:55,785][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:51:56,284][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:51:56,782][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:51:57,287][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:51:57,784][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:51:58,283][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:51:58,781][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:51:59,280][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:51:59,783][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:52:00,281][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:52:00,779][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:52:01,278][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:52:01,778][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:52:02,277][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:52:02,777][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:52:03,276][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:52:03,779][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:52:04,280][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:52:04,780][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:52:05,279][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:52:05,779][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:52:06,280][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:52:06,779][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:52:07,279][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:52:07,779][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:52:08,278][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:52:08,777][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:52:09,284][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:52:09,785][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:52:10,288][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:52:10,789][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:52:11,287][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:52:11,787][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:52:12,288][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:52:12,791][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:52:13,293][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:52:13,797][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:52:14,299][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:52:14,798][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:52:15,300][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10819 tokens.
[2025-11-12 23:52:15,963][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-12 23:52:16,715][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:52:16,717][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:52:16,718][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:52:17,604][__main__][INFO] - Iteration 116 took 51s (30.29% Gen, 67.98% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 3m 36s. Estimated total time: 42h 46m 12s. Time estimates for 10 more iterations: 8m 33s, 100 more iterations: 1h 25m 32s, 500 more iterations: 7h 7m 42s.
[2025-11-12 23:52:17,607][__main__][INFO] - Starting iteration 116.
[2025-11-12 23:52:18,100][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:52:18,101][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:52:30,763][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:52:34,932][__main__][INFO] - Number of regex retries in iteration 116: 1
[2025-11-12 23:52:34,933][__main__][INFO] - agents played in iteration 116 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:52:35,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:52:35,761][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:52:35,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:52:35,808][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:52:35,808][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:52:35,809][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:52:36,424][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:52:36,882][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:52:37,386][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:52:37,886][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:52:38,385][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:52:38,883][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:52:39,378][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:52:39,877][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:52:40,376][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:52:40,871][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:52:41,369][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:52:41,867][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:52:42,366][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:52:42,866][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:52:43,362][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:52:43,862][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:52:44,361][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:52:44,865][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:52:45,368][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:52:45,873][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:52:46,377][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:52:46,883][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:52:47,388][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:52:47,891][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:52:48,397][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:52:48,896][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:52:49,397][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:52:49,902][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:52:50,404][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:52:50,904][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:52:51,402][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:52:51,899][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:52:52,400][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:52:52,898][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:52:53,398][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:52:53,897][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:52:54,396][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:52:54,896][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:52:55,397][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:52:55,897][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:52:56,399][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:52:56,899][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:52:57,399][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:52:57,899][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:52:58,399][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:52:58,902][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:52:59,402][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:52:59,903][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:53:00,402][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:53:00,905][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:53:01,406][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:53:01,910][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:53:02,418][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:53:02,931][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:53:03,435][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:53:03,951][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:53:04,451][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:53:04,948][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:53:05,449][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:53:05,946][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:53:06,443][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:53:06,942][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:53:07,439][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:53:07,949][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:53:08,446][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10582 tokens.
[2025-11-12 23:53:09,145][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-12 23:53:10,036][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:53:10,037][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:53:10,053][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:53:11,118][__main__][INFO] - Iteration 117 took 53s (31.75% Gen, 66.24% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 27m 25s. Estimated total time: 44h 10m 55s. Time estimates for 10 more iterations: 8m 50s, 100 more iterations: 1h 28m 21s, 500 more iterations: 7h 21m 49s.
[2025-11-12 23:53:11,120][__main__][INFO] - Starting iteration 117.
[2025-11-12 23:53:11,632][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:53:11,633][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:53:28,197][__main__][INFO] - Number of regex retries in iteration 117: 0
[2025-11-12 23:53:28,198][__main__][INFO] - agents played in iteration 117 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:53:29,033][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:53:29,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:53:29,087][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:53:29,110][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:53:29,110][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:53:29,112][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:53:29,711][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:53:30,167][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:53:30,668][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:53:31,163][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:53:31,665][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:53:32,162][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:53:32,661][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:53:33,159][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:53:33,656][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:53:34,165][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:53:34,663][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:53:35,164][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:53:35,663][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:53:36,161][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:53:36,668][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:53:37,165][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:53:37,667][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:53:38,167][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:53:38,670][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:53:39,167][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:53:39,667][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:53:40,168][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:53:40,671][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:53:41,168][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:53:41,669][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:53:42,168][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:53:42,665][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:53:43,168][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:53:43,667][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:53:44,165][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:53:44,664][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:53:45,162][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:53:45,660][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:53:46,164][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:53:46,665][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:53:47,175][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:53:47,701][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:53:48,203][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:53:48,703][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:53:49,205][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:53:49,707][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:53:50,208][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:53:50,705][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:53:51,204][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:53:51,704][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:53:52,206][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:53:52,705][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:53:53,204][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:53:53,705][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:53:54,206][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:53:54,706][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:53:55,209][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:53:55,708][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:53:56,209][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:53:56,709][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:53:57,211][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:53:57,713][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:53:58,212][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:53:58,714][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:53:59,216][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:53:59,717][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:54:00,217][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:54:00,717][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:54:01,217][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:54:01,728][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10831 tokens.
[2025-11-12 23:54:02,423][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.40%, ΔTime: 00:00:32
[2025-11-12 23:54:03,160][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:54:03,162][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:54:03,165][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:54:04,069][__main__][INFO] - Iteration 118 took 52s (31.59% Gen, 66.68% Train). Generation: 16s, Training: 34s. Estimated remaining time: 41h 57m 29s. Estimated total time: 43h 41m 51s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 23s, 500 more iterations: 7h 16m 58s.
[2025-11-12 23:54:04,071][__main__][INFO] - Starting iteration 118.
[2025-11-12 23:54:04,537][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:54:04,537][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:54:07,532][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:54:07,581][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:54:20,496][__main__][INFO] - Number of regex retries in iteration 118: 2
[2025-11-12 23:54:20,496][__main__][INFO] - agents played in iteration 118 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:54:21,361][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:54:21,389][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:54:21,414][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:54:21,437][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:54:21,437][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:54:21,438][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:54:22,040][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:54:22,500][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:54:23,003][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:54:23,502][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:54:24,001][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:54:24,502][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:54:25,003][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:54:25,503][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:54:26,003][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:54:26,505][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:54:27,010][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:54:27,508][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:54:28,009][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:54:28,509][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:54:29,010][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:54:29,508][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:54:30,009][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:54:30,510][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:54:31,008][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:54:31,507][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:54:32,004][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:54:32,503][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:54:33,005][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:54:33,504][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:54:34,002][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:54:34,501][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:54:34,999][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:54:35,500][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:54:35,999][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:54:36,500][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:54:36,999][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:54:37,497][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:54:37,999][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:54:38,496][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:54:38,994][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:54:39,493][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:54:39,989][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:54:40,489][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:54:40,987][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:54:41,485][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:54:41,989][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:54:42,489][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:54:42,989][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:54:43,492][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:54:43,996][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:54:44,496][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:54:44,995][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:54:45,494][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:54:45,999][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:54:46,500][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:54:47,009][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:54:47,511][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:54:48,011][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:54:48,531][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:54:49,032][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:54:49,532][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:54:50,044][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:54:50,543][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:54:51,044][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:54:51,543][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:54:52,042][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:54:52,544][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:54:53,044][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:54:53,545][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:54:54,047][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10770 tokens.
[2025-11-12 23:54:54,727][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-12 23:54:55,503][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:54:55,504][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:54:55,506][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:54:56,515][__main__][INFO] - Iteration 119 took 51s (30.70% Gen, 67.35% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 33m 43s. Estimated total time: 43h 18m 57s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 37s, 500 more iterations: 7h 13m 9s.
[2025-11-12 23:54:56,517][__main__][INFO] - Starting iteration 119.
[2025-11-12 23:54:56,992][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:54:56,992][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:55:02,161][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:55:10,144][mllm.models.large_language_model_local][WARNING] - Response Given the current item values and the symmetric nature of the game where both Alice and Bob value the items in a way that maximizes their own utility, it seems optimal to propose a split that ensures both parties receive items they value. Here, since both Alice and Bob value the hats, books, and balls relatively evenly, the best strategy is likely to be an even split.

Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:55:13,358][__main__][INFO] - Number of regex retries in iteration 119: 2
[2025-11-12 23:55:13,358][__main__][INFO] - agents played in iteration 119 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:55:14,163][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:55:14,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:55:14,209][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:55:14,231][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:55:14,231][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:55:14,232][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:55:14,843][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:55:15,298][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:55:15,798][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:55:16,299][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:55:16,801][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:55:17,301][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:55:17,802][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:55:18,301][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:55:18,806][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:55:19,310][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:55:19,809][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:55:20,310][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:55:20,808][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:55:21,311][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:55:21,811][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:55:22,311][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:55:22,814][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:55:23,310][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:55:23,812][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:55:24,312][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:55:24,810][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:55:25,304][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:55:25,800][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:55:26,297][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:55:26,802][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:55:27,301][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:55:27,798][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:55:28,297][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:55:28,796][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:55:29,300][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:55:29,797][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:55:30,297][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:55:30,800][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:55:31,301][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:55:31,800][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:55:32,306][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:55:32,806][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:55:33,325][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:55:33,828][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:55:34,331][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:55:34,837][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:55:35,342][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:55:35,841][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:55:36,343][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:55:36,841][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:55:37,345][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:55:37,846][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:55:38,347][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:55:38,852][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:55:39,353][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:55:39,855][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:55:40,356][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:55:40,858][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:55:41,362][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:55:41,866][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:55:42,367][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:55:42,872][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:55:43,373][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:55:43,884][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:55:44,388][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:55:44,888][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:55:45,400][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:55:45,900][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:55:46,403][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:55:46,899][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10844 tokens.
[2025-11-12 23:55:47,550][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.39%, ΔTime: 00:00:32
[2025-11-12 23:55:48,317][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:55:48,319][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:55:48,321][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:55:49,341][__main__][INFO] - Iteration 120 took 52s (31.26% Gen, 66.78% Train). Generation: 16s, Training: 34s. Estimated remaining time: 41h 51m 24s. Estimated total time: 43h 37m 32s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 15s, 500 more iterations: 7h 16m 15s.
[2025-11-12 23:55:49,343][__main__][INFO] - Starting iteration 120.
[2025-11-12 23:55:49,846][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-12 23:55:49,847][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:55:57,716][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:56:05,241][__main__][INFO] - Number of regex retries in iteration 120: 1
[2025-11-12 23:56:05,242][__main__][INFO] - agents played in iteration 120 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:56:06,107][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:56:06,140][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:56:06,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:56:06,192][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:56:06,193][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:56:06,194][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:56:06,794][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:56:07,246][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:56:07,765][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:56:08,266][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:56:08,766][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:56:09,275][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:56:09,774][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:56:10,295][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:56:10,793][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:56:11,293][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:56:11,794][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:56:12,294][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:56:12,798][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:56:13,299][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:56:13,796][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:56:14,297][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:56:14,794][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:56:15,296][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:56:15,795][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:56:16,293][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:56:16,800][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:56:17,300][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:56:17,799][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:56:18,299][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:56:18,798][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:56:19,304][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:56:19,801][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:56:20,299][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:56:20,805][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:56:21,303][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:56:21,800][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:56:22,297][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:56:22,794][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:56:23,298][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:56:23,797][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:56:24,298][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:56:24,796][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:56:25,294][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:56:25,796][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:56:26,297][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:56:26,800][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:56:27,307][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:56:27,805][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:56:28,303][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:56:28,798][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:56:29,295][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:56:29,798][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:56:30,297][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:56:30,798][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:56:31,306][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:56:31,806][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:56:32,308][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:56:32,809][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:56:33,309][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:56:33,816][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:56:34,320][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:56:34,821][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:56:35,324][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:56:35,824][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:56:36,324][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:56:36,823][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:56:37,323][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:56:37,828][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:56:38,325][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:56:38,827][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10861 tokens.
[2025-11-12 23:56:39,482][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-12 23:56:40,239][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:56:40,241][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:56:40,243][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:56:42,231][__main__][INFO] - Iteration 121 took 52s (29.39% Gen, 66.81% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 52m 16s. Estimated total time: 43h 39m 16s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 18s, 500 more iterations: 7h 16m 32s.
[2025-11-12 23:56:42,233][__main__][INFO] - Starting iteration 121.
[2025-11-12 23:56:42,720][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-12 23:56:42,720][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:56:56,590][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:56:59,167][__main__][INFO] - Number of regex retries in iteration 121: 1
[2025-11-12 23:56:59,168][__main__][INFO] - agents played in iteration 121 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:57:00,045][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:57:00,070][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:57:00,095][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:57:00,118][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:57:00,119][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:57:00,119][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:57:00,772][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:57:01,229][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:57:01,732][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:57:02,228][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:57:02,732][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:57:03,232][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:57:03,732][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:57:04,233][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:57:04,731][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:57:05,228][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:57:05,728][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:57:06,225][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:57:06,723][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:57:07,220][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:57:07,719][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:57:08,213][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:57:08,711][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:57:09,213][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:57:09,713][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:57:10,213][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:57:10,716][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:57:11,215][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:57:11,713][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:57:12,209][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:57:12,708][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:57:13,210][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:57:13,704][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:57:14,201][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:57:14,700][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:57:15,200][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:57:15,700][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:57:16,197][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:57:16,692][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:57:17,209][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:57:17,710][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:57:18,211][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:57:18,717][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:57:19,215][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:57:19,716][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:57:20,213][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:57:20,709][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:57:21,216][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:57:21,717][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:57:22,219][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:57:22,717][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:57:23,218][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:57:23,721][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:57:24,222][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:57:24,725][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:57:25,227][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:57:25,728][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:57:26,231][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:57:26,733][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:57:27,238][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:57:27,744][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:57:28,244][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:57:28,742][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:57:29,243][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:57:29,741][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:57:30,241][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:57:30,740][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:57:31,238][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:57:31,735][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:57:32,234][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:57:32,731][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10823 tokens.
[2025-11-12 23:57:33,402][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-12 23:57:34,159][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:57:34,161][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:57:34,162][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:57:35,085][__main__][INFO] - Iteration 122 took 52s (31.41% Gen, 66.83% Train). Generation: 16s, Training: 34s. Estimated remaining time: 41h 50m 24s. Estimated total time: 43h 38m 17s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 16s, 500 more iterations: 7h 16m 22s.
[2025-11-12 23:57:35,087][__main__][INFO] - Starting iteration 122.
[2025-11-12 23:57:35,852][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-12 23:57:35,853][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:57:42,279][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given Bob's per-item values, he will likely pursue items that have high value for him, such as books and balls. Therefore, proposing to take all 10 hats is strategically sound, as it maximizes the items you value highly while minimizing the risk of proportional allocation due to exceeding the total item quantity. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:57:50,952][__main__][INFO] - Number of regex retries in iteration 122: 1
[2025-11-12 23:57:50,953][__main__][INFO] - agents played in iteration 122 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:57:51,852][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:57:51,879][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:57:51,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:57:51,929][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:57:51,929][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:57:51,931][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:57:52,569][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:57:53,043][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:57:53,548][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:57:54,048][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:57:54,548][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:57:55,047][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:57:55,551][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:57:56,050][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:57:56,549][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:57:57,048][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:57:57,551][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:57:58,049][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:57:58,550][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:57:59,048][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:57:59,548][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:58:00,045][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:58:00,544][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:58:01,045][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:58:01,543][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:58:02,041][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:58:02,539][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:58:03,038][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:58:03,537][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:58:04,035][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:58:04,535][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:58:05,036][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:58:05,535][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:58:06,034][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:58:06,534][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:58:07,033][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:58:07,533][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:58:08,033][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:58:08,532][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:58:09,035][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:58:09,539][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:58:10,040][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:58:10,541][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:58:11,039][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:58:11,539][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:58:12,038][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:58:12,536][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:58:13,037][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:58:13,535][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:58:14,035][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:58:14,548][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:58:15,058][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:58:15,565][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:58:16,064][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:58:16,563][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:58:17,077][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:58:17,578][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:58:18,085][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:58:18,585][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:58:19,084][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:58:19,583][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:58:20,082][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:58:20,581][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:58:21,079][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:58:21,577][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:58:22,092][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:58:22,591][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:58:23,088][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:58:23,592][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:58:24,088][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:58:24,596][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10873 tokens.
[2025-11-12 23:58:25,254][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-12 23:58:26,014][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:58:26,015][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:58:26,017][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:58:26,978][__main__][INFO] - Iteration 123 took 51s (29.54% Gen, 68.58% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 47m 33s. Estimated total time: 42h 36m 18s. Time estimates for 10 more iterations: 8m 31s, 100 more iterations: 1h 25m 12s, 500 more iterations: 7h 6m 3s.
[2025-11-12 23:58:26,980][__main__][INFO] - Starting iteration 123.
[2025-11-12 23:58:27,468][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-12 23:58:27,468][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:58:34,992][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-12 23:58:43,271][__main__][INFO] - Number of regex retries in iteration 123: 1
[2025-11-12 23:58:43,272][__main__][INFO] - agents played in iteration 123 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:58:44,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:58:44,088][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:58:44,114][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:58:44,137][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:58:44,137][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:58:44,138][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:58:44,779][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:58:45,239][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:58:45,745][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:58:46,243][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:58:46,745][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:58:47,245][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:58:47,745][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:58:48,246][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:58:48,744][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:58:49,241][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:58:49,741][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:58:50,238][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:58:50,739][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:58:51,237][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:58:51,737][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:58:52,236][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:58:52,735][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:58:53,234][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:58:53,735][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:58:54,232][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:58:54,731][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:58:55,232][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:58:55,732][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:58:56,232][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:58:56,736][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:58:57,241][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:58:57,743][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:58:58,243][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:58:58,745][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:58:59,247][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:58:59,744][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:59:00,243][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:59:00,741][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:59:01,238][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:59:01,737][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:59:02,235][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:59:02,738][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:59:03,236][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:59:03,735][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:59:04,235][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:59:04,741][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:59:05,248][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:59:05,749][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:59:06,250][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:59:06,754][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:59:07,254][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-12 23:59:07,757][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-12 23:59:08,258][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-12 23:59:08,759][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-12 23:59:09,262][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-12 23:59:09,760][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-12 23:59:10,261][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-12 23:59:10,760][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-12 23:59:11,263][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-12 23:59:11,763][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-12 23:59:12,261][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-12 23:59:12,759][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-12 23:59:13,262][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-12 23:59:13,759][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-12 23:59:14,266][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-12 23:59:14,767][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-12 23:59:15,265][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-12 23:59:15,768][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-12 23:59:16,266][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-12 23:59:16,765][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10843 tokens.
[2025-11-12 23:59:17,417][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-12 23:59:18,188][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-12 23:59:18,190][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-12 23:59:18,192][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-12 23:59:19,333][__main__][INFO] - Iteration 124 took 51s (30.47% Gen, 67.33% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 23m 39s. Estimated total time: 43h 13m 17s. Time estimates for 10 more iterations: 8m 38s, 100 more iterations: 1h 26m 26s, 500 more iterations: 7h 12m 12s.
[2025-11-12 23:59:19,335][__main__][INFO] - Starting iteration 124.
[2025-11-12 23:59:19,829][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-12 23:59:19,830][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-12 23:59:35,810][__main__][INFO] - Number of regex retries in iteration 124: 0
[2025-11-12 23:59:35,811][__main__][INFO] - agents played in iteration 124 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-12 23:59:36,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:59:36,727][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:59:36,749][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:59:36,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-12 23:59:36,772][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-12 23:59:36,773][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-12 23:59:37,400][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-12 23:59:37,864][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-12 23:59:38,363][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-12 23:59:38,860][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-12 23:59:39,364][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-12 23:59:39,862][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-12 23:59:40,360][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-12 23:59:40,858][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-12 23:59:41,358][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-12 23:59:41,861][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-12 23:59:42,355][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-12 23:59:42,853][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-12 23:59:43,355][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-12 23:59:43,855][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-12 23:59:44,353][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-12 23:59:44,852][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-12 23:59:45,352][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-12 23:59:45,853][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-12 23:59:46,352][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-12 23:59:46,851][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-12 23:59:47,347][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-12 23:59:47,847][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-12 23:59:48,348][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-12 23:59:48,852][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-12 23:59:49,353][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-12 23:59:49,854][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-12 23:59:50,349][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-12 23:59:50,845][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-12 23:59:51,341][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-12 23:59:51,837][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-12 23:59:52,340][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-12 23:59:52,836][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-12 23:59:53,333][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-12 23:59:53,841][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-12 23:59:54,339][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-12 23:59:54,836][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-12 23:59:55,335][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-12 23:59:55,836][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-12 23:59:56,346][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-12 23:59:56,846][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-12 23:59:57,347][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-12 23:59:57,850][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-12 23:59:58,348][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-12 23:59:58,848][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-12 23:59:59,347][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-12 23:59:59,845][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:00:00,346][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:00:00,842][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:00:01,339][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:00:01,866][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:00:02,364][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:00:02,861][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:00:03,364][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:00:03,862][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:00:04,363][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:00:04,861][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:00:05,357][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:00:05,858][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:00:06,357][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:00:06,852][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:00:07,351][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:00:07,852][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:00:08,349][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:00:08,849][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:00:09,348][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10823 tokens.
[2025-11-13 00:00:10,011][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 00:00:10,761][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:00:10,763][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:00:10,764][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:00:11,699][__main__][INFO] - Iteration 125 took 51s (30.81% Gen, 67.39% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 23m 2s. Estimated total time: 43h 13m 32s. Time estimates for 10 more iterations: 8m 38s, 100 more iterations: 1h 26m 27s, 500 more iterations: 7h 12m 15s.
[2025-11-13 00:00:11,702][__main__][INFO] - Starting iteration 125.
[2025-11-13 00:00:12,187][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 00:00:12,188][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:00:28,925][__main__][INFO] - Number of regex retries in iteration 125: 0
[2025-11-13 00:00:28,926][__main__][INFO] - agents played in iteration 125 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:00:29,781][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:00:29,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:00:29,825][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:00:29,847][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:00:29,847][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:00:29,848][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:00:30,458][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:00:30,923][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:00:31,425][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:00:31,922][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:00:32,423][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:00:32,922][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:00:33,426][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:00:33,925][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:00:34,423][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:00:34,941][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:00:35,442][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:00:35,942][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:00:36,443][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:00:36,942][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:00:37,443][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:00:37,942][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:00:38,443][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:00:38,948][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:00:39,447][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:00:39,947][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:00:40,452][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:00:40,953][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:00:41,451][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:00:41,947][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:00:42,442][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:00:42,951][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:00:43,448][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:00:43,944][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:00:44,443][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:00:44,939][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:00:45,436][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:00:45,934][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:00:46,435][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:00:46,948][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:00:47,449][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:00:47,949][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:00:48,450][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:00:48,951][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:00:49,454][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:00:49,956][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:00:50,457][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:00:50,960][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:00:51,462][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:00:51,963][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:00:52,465][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:00:52,962][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:00:53,463][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:00:53,961][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:00:54,459][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:00:54,958][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:00:55,458][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:00:55,958][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:00:56,457][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:00:56,952][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:00:57,452][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:00:57,952][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:00:58,451][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:00:58,956][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:00:59,456][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:00:59,954][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:01:00,452][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:01:00,948][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:01:01,452][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:01:01,951][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:01:02,450][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10826 tokens.
[2025-11-13 00:01:03,100][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 00:01:03,858][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:01:03,860][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:01:03,862][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:01:05,050][__main__][INFO] - Iteration 126 took 52s (31.66% Gen, 66.09% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 11m 48s. Estimated total time: 44h 3m 11s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 6s, 500 more iterations: 7h 20m 31s.
[2025-11-13 00:01:05,053][__main__][INFO] - Starting iteration 126.
[2025-11-13 00:01:05,530][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 00:01:05,530][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:01:13,800][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:01:20,215][__main__][INFO] - Number of regex retries in iteration 126: 1
[2025-11-13 00:01:20,216][__main__][INFO] - agents played in iteration 126 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:01:21,047][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:01:21,075][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:01:21,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:01:21,125][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:01:21,125][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:01:21,126][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:01:21,746][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:01:22,204][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:01:22,721][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:01:23,220][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:01:23,724][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:01:24,222][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:01:24,719][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:01:25,228][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:01:25,727][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:01:26,240][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:01:26,743][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:01:27,243][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:01:27,769][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:01:28,274][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:01:28,773][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:01:29,279][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:01:29,780][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:01:30,279][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:01:30,778][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:01:31,277][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:01:31,776][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:01:32,278][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:01:32,778][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:01:33,276][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:01:33,772][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:01:34,272][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:01:34,776][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:01:35,277][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:01:35,781][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:01:36,281][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:01:36,782][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:01:37,281][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:01:37,782][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:01:38,286][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:01:38,785][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:01:39,287][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:01:39,788][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:01:40,288][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:01:40,792][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:01:41,292][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:01:41,792][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:01:42,293][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:01:42,794][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:01:43,297][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:01:43,799][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:01:44,301][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:01:44,804][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:01:45,304][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:01:45,805][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:01:46,306][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:01:46,810][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:01:47,315][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:01:47,816][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:01:48,315][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:01:48,828][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:01:49,331][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:01:49,845][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:01:50,345][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:01:50,843][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:01:51,358][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:01:51,857][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:01:52,361][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:01:52,863][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:01:53,365][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:01:53,868][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10849 tokens.
[2025-11-13 00:01:54,517][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 00:01:55,305][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:01:55,306][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:01:55,308][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:01:56,250][__main__][INFO] - Iteration 127 took 50s (28.95% Gen, 69.19% Train). Generation: 14s, Training: 35s. Estimated remaining time: 40h 23m 46s. Estimated total time: 42h 16m 0s. Time estimates for 10 more iterations: 8m 27s, 100 more iterations: 1h 24m 32s, 500 more iterations: 7h 2m 40s.
[2025-11-13 00:01:56,252][__main__][INFO] - Starting iteration 127.
[2025-11-13 00:01:56,724][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 00:01:56,725][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:02:00,934][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:02:12,124][__main__][INFO] - Number of regex retries in iteration 127: 1
[2025-11-13 00:02:12,125][__main__][INFO] - agents played in iteration 127 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:02:12,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:02:12,934][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:02:12,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:02:12,984][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:02:12,985][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:02:12,986][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:02:13,617][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:02:14,074][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:02:14,573][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:02:15,077][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:02:15,577][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:02:16,076][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:02:16,575][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:02:17,071][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:02:17,572][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:02:18,075][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:02:18,574][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:02:19,072][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:02:19,568][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:02:20,066][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:02:20,583][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:02:21,087][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:02:21,590][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:02:22,094][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:02:22,592][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:02:23,089][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:02:23,588][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:02:24,086][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:02:24,583][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:02:25,080][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:02:25,580][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:02:26,079][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:02:26,577][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:02:27,076][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:02:27,575][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:02:28,076][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:02:28,571][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:02:29,068][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:02:29,575][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:02:30,077][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:02:30,583][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:02:31,085][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:02:31,587][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:02:32,090][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:02:32,591][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:02:33,093][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:02:33,594][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:02:34,094][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:02:34,594][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:02:35,094][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:02:35,593][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:02:36,093][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:02:36,591][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:02:37,088][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:02:37,588][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:02:38,087][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:02:38,586][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:02:39,086][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:02:39,586][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:02:40,086][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:02:40,584][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:02:41,083][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:02:41,582][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:02:42,082][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:02:42,587][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:02:43,087][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:02:43,587][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:02:44,089][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:02:44,590][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:02:45,092][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:02:45,592][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10823 tokens.
[2025-11-13 00:02:46,280][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.38%, ΔTime: 00:00:32
[2025-11-13 00:02:47,042][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:02:47,044][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:02:47,045][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:02:48,132][__main__][INFO] - Iteration 128 took 51s (29.96% Gen, 67.93% Train). Generation: 15s, Training: 34s. Estimated remaining time: 40h 57m 20s. Estimated total time: 42h 50m 27s. Time estimates for 10 more iterations: 8m 34s, 100 more iterations: 1h 25m 40s, 500 more iterations: 7h 8m 24s.
[2025-11-13 00:02:48,134][__main__][INFO] - Starting iteration 128.
[2025-11-13 00:02:48,623][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 00:02:48,624][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:02:52,483][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:02:58,914][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:03:05,395][__main__][INFO] - Number of regex retries in iteration 128: 2
[2025-11-13 00:03:05,395][__main__][INFO] - agents played in iteration 128 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:03:06,207][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:03:06,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:03:06,259][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:03:06,282][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:03:06,283][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:03:06,284][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:03:06,907][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:03:07,360][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:03:07,865][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:03:08,365][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:03:08,863][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:03:09,364][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:03:09,862][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:03:10,364][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:03:10,862][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:03:11,360][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:03:11,858][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:03:12,359][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:03:12,869][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:03:13,368][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:03:13,865][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:03:14,363][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:03:14,858][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:03:15,362][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:03:15,858][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:03:16,354][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:03:16,851][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:03:17,346][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:03:17,845][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:03:18,348][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:03:18,846][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:03:19,357][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:03:19,854][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:03:20,354][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:03:20,856][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:03:21,360][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:03:21,863][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:03:22,365][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:03:22,866][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:03:23,373][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:03:23,876][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:03:24,377][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:03:24,878][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:03:25,379][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:03:25,879][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:03:26,380][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:03:26,881][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:03:27,384][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:03:27,885][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:03:28,384][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:03:28,884][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:03:29,384][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:03:29,884][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:03:30,384][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:03:30,884][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:03:31,384][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:03:31,883][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:03:32,383][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:03:32,882][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:03:33,382][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:03:33,883][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:03:34,380][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:03:34,882][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:03:35,408][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:03:35,981][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:03:36,486][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:03:36,988][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:03:37,489][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:03:37,987][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:03:38,486][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:03:38,989][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10821 tokens.
[2025-11-13 00:03:39,674][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.29%, Current % of VRAM taken: 60.55%, Block Peak % of device VRAM: 62.42%, ΔTime: 00:00:32
[2025-11-13 00:03:40,452][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:03:40,454][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:03:40,456][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:03:41,492][__main__][INFO] - Iteration 129 took 52s (31.72% Gen, 66.31% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 9m 29s. Estimated total time: 44h 3m 29s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 6s, 500 more iterations: 7h 20m 34s.
[2025-11-13 00:03:41,495][__main__][INFO] - Starting iteration 129.
[2025-11-13 00:03:41,980][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 00:03:41,980][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:03:45,594][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:03:50,517][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:03:58,435][__main__][INFO] - Number of regex retries in iteration 129: 2
[2025-11-13 00:03:58,436][__main__][INFO] - agents played in iteration 129 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:03:59,281][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:03:59,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:03:59,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:03:59,355][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:03:59,356][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:03:59,356][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:03:59,996][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:04:00,456][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:04:00,964][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:04:01,480][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:04:01,982][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:04:02,483][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:04:02,982][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:04:03,480][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:04:03,985][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:04:04,486][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:04:04,986][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:04:05,486][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:04:05,985][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:04:06,485][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:04:06,983][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:04:07,479][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:04:07,982][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:04:08,481][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:04:08,984][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:04:09,484][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:04:09,985][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:04:10,486][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:04:10,987][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:04:11,488][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:04:11,990][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:04:12,489][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:04:12,990][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:04:13,494][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:04:13,996][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:04:14,500][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:04:15,001][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:04:15,503][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:04:16,005][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:04:16,505][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:04:17,005][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:04:17,506][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:04:18,007][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:04:18,510][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:04:19,009][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:04:19,504][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:04:20,005][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:04:20,506][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:04:21,009][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:04:21,505][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:04:22,006][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:04:22,505][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:04:23,003][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:04:23,505][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:04:24,007][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:04:24,510][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:04:25,012][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:04:25,513][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:04:26,012][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:04:26,512][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:04:27,013][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:04:27,516][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:04:28,022][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:04:28,523][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:04:29,022][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:04:29,521][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:04:30,026][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:04:30,526][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:04:31,027][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:04:31,531][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:04:32,032][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10837 tokens.
[2025-11-13 00:04:32,684][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 00:04:33,442][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:04:33,444][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:04:33,446][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:04:34,393][__main__][INFO] - Iteration 130 took 52s (31.39% Gen, 66.80% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 45m 48s. Estimated total time: 43h 40m 40s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 21s, 500 more iterations: 7h 16m 46s.
[2025-11-13 00:04:34,395][__main__][INFO] - Starting iteration 130.
[2025-11-13 00:04:34,878][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 00:04:34,878][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:04:49,770][__main__][INFO] - Number of regex retries in iteration 130: 0
[2025-11-13 00:04:49,770][__main__][INFO] - agents played in iteration 130 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:04:50,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:04:50,624][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:04:50,650][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:04:50,673][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:04:50,673][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:04:50,675][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:04:51,308][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:04:51,765][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:04:52,267][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:04:52,763][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:04:53,264][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:04:53,775][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:04:54,272][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:04:54,768][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:04:55,282][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:04:55,839][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:04:56,345][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:04:56,846][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:04:57,346][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:04:57,856][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:04:58,358][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:04:58,854][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:04:59,359][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:04:59,858][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:05:00,366][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:05:00,866][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:05:01,365][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:05:01,867][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:05:02,367][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:05:02,867][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:05:03,364][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:05:03,868][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:05:04,385][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:05:04,888][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:05:05,392][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:05:05,890][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:05:06,391][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:05:06,894][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:05:07,394][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:05:07,893][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:05:08,396][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:05:08,897][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:05:09,399][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:05:09,900][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:05:10,402][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:05:10,904][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:05:11,404][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:05:11,905][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:05:12,405][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:05:12,904][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:05:13,403][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:05:13,903][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:05:14,403][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:05:14,905][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:05:15,404][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:05:15,905][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:05:16,403][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:05:16,903][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:05:17,405][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:05:17,905][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:05:18,406][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:05:18,907][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:05:19,404][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:05:19,903][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:05:20,401][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:05:20,902][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:05:21,404][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:05:21,903][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:05:22,401][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:05:22,902][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:05:23,402][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10855 tokens.
[2025-11-13 00:05:24,069][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 00:05:24,825][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:05:24,826][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:05:24,829][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:05:26,667][__main__][INFO] - Iteration 131 took 51s (28.75% Gen, 67.69% Train). Generation: 14s, Training: 35s. Estimated remaining time: 41h 13m 46s. Estimated total time: 43h 9m 31s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 19s, 500 more iterations: 7h 11m 35s.
[2025-11-13 00:05:26,670][__main__][INFO] - Starting iteration 131.
[2025-11-13 00:05:27,136][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:05:27,137][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:05:30,458][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 5 hats, 5 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:05:41,548][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:05:42,474][__main__][INFO] - Number of regex retries in iteration 131: 2
[2025-11-13 00:05:42,475][__main__][INFO] - agents played in iteration 131 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:05:43,300][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:05:43,329][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:05:43,355][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:05:43,378][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:05:43,379][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:05:43,380][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:05:44,017][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:05:44,477][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:05:44,983][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:05:45,482][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:05:45,986][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:05:46,495][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:05:46,996][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:05:47,496][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:05:47,996][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:05:48,501][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:05:49,001][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:05:49,501][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:05:50,002][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:05:50,501][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:05:50,999][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:05:51,506][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:05:52,015][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:05:52,516][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:05:53,017][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:05:53,515][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:05:54,014][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:05:54,517][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:05:55,016][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:05:55,517][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:05:56,019][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:05:56,516][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:05:57,015][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:05:57,513][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:05:58,012][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:05:58,514][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:05:59,016][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:05:59,520][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:06:00,020][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:06:00,522][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:06:01,025][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:06:01,523][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:06:02,024][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:06:02,525][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:06:03,024][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:06:03,524][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:06:04,024][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:06:04,527][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:06:05,026][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:06:05,526][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:06:06,027][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:06:06,525][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:06:07,026][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:06:07,529][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:06:08,028][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:06:08,527][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:06:09,027][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:06:09,527][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:06:10,029][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:06:10,527][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:06:11,025][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:06:11,523][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:06:12,021][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:06:12,523][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:06:13,021][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:06:13,521][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:06:14,019][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:06:14,519][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:06:15,020][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:06:15,519][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:06:16,020][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10829 tokens.
[2025-11-13 00:06:16,677][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.39%, ΔTime: 00:00:32
[2025-11-13 00:06:17,460][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:06:17,461][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:06:17,463][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:06:18,651][__main__][INFO] - Iteration 132 took 51s (29.77% Gen, 67.92% Train). Generation: 15s, Training: 34s. Estimated remaining time: 40h 59m 8s. Estimated total time: 42h 55m 45s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 51s, 500 more iterations: 7h 9m 17s.
[2025-11-13 00:06:18,653][__main__][INFO] - Starting iteration 132.
[2025-11-13 00:06:19,145][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:06:19,145][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:06:35,163][__main__][INFO] - Number of regex retries in iteration 132: 0
[2025-11-13 00:06:35,163][__main__][INFO] - agents played in iteration 132 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:06:36,008][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:06:36,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:06:36,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:06:36,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:06:36,085][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:06:36,085][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:06:36,759][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:06:37,215][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:06:37,719][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:06:38,216][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:06:38,714][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:06:39,211][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:06:39,708][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:06:40,206][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:06:40,702][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:06:41,200][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:06:41,699][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:06:42,199][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:06:42,711][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:06:43,210][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:06:43,711][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:06:44,214][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:06:44,712][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:06:45,217][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:06:45,717][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:06:46,218][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:06:46,721][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:06:47,221][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:06:47,720][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:06:48,224][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:06:48,721][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:06:49,219][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:06:49,715][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:06:50,212][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:06:50,717][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:06:51,219][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:06:51,723][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:06:52,222][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:06:52,721][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:06:53,223][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:06:53,738][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:06:54,238][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:06:54,741][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:06:55,239][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:06:55,739][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:06:56,238][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:06:56,738][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:06:57,241][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:06:57,738][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:06:58,236][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:06:58,731][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:06:59,231][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:06:59,733][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:07:00,235][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:07:00,734][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:07:01,234][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:07:01,731][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:07:02,232][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:07:02,731][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:07:03,229][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:07:03,731][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:07:04,228][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:07:04,726][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:07:05,226][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:07:05,725][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:07:06,224][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:07:06,722][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:07:07,220][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:07:07,722][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:07:08,220][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:07:08,722][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10842 tokens.
[2025-11-13 00:07:09,407][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 00:07:10,324][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:07:10,325][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:07:10,342][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:07:11,606][__main__][INFO] - Iteration 133 took 52s (30.53% Gen, 67.05% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 45m 35s. Estimated total time: 43h 43m 5s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 26s, 500 more iterations: 7h 17m 10s.
[2025-11-13 00:07:11,608][__main__][INFO] - Starting iteration 133.
[2025-11-13 00:07:12,111][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:07:12,111][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:07:19,194][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:07:29,057][__main__][INFO] - Number of regex retries in iteration 133: 1
[2025-11-13 00:07:29,058][__main__][INFO] - agents played in iteration 133 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:07:29,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:07:29,886][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:07:29,908][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:07:29,943][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:07:29,944][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:07:29,945][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:07:30,543][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:07:31,001][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:07:31,515][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:07:32,014][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:07:32,511][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:07:33,021][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:07:33,524][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:07:34,029][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:07:34,531][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:07:35,031][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:07:35,535][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:07:36,034][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:07:36,535][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:07:37,035][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:07:37,538][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:07:38,046][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:07:38,543][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:07:39,044][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:07:39,545][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:07:40,043][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:07:40,542][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:07:41,039][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:07:41,542][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:07:42,042][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:07:42,541][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:07:43,042][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:07:43,543][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:07:44,042][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:07:44,546][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:07:45,048][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:07:45,549][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:07:46,053][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:07:46,553][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:07:47,052][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:07:47,553][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:07:48,057][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:07:48,561][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:07:49,061][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:07:49,561][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:07:50,071][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:07:50,569][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:07:51,076][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:07:51,576][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:07:52,073][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:07:52,584][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:07:53,082][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:07:53,589][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:07:54,086][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:07:54,584][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:07:55,102][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:07:55,604][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:07:56,105][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:07:56,604][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:07:57,102][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:07:57,613][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:07:58,110][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:07:58,609][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:07:59,107][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:07:59,606][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:08:00,114][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:08:00,614][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:08:01,112][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:08:01,620][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:08:02,121][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:08:02,625][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10818 tokens.
[2025-11-13 00:08:03,321][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.48%, ΔTime: 00:00:32
[2025-11-13 00:08:04,073][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:08:04,076][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:08:04,078][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:08:04,945][__main__][INFO] - Iteration 134 took 52s (32.07% Gen, 66.28% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 3m 23s. Estimated total time: 44h 1m 46s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 3s, 500 more iterations: 7h 20m 17s.
[2025-11-13 00:08:04,948][__main__][INFO] - Starting iteration 134.
[2025-11-13 00:08:05,438][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:08:05,438][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:08:21,584][__main__][INFO] - Number of regex retries in iteration 134: 0
[2025-11-13 00:08:21,584][__main__][INFO] - agents played in iteration 134 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:08:22,474][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:08:22,500][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:08:22,526][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:08:22,549][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:08:22,549][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:08:22,550][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:08:23,212][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:08:23,669][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:08:24,186][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:08:24,685][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:08:25,202][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:08:25,704][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:08:26,204][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:08:26,715][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:08:27,213][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:08:27,717][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:08:28,213][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:08:28,710][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:08:29,212][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:08:29,710][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:08:30,209][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:08:30,709][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:08:31,208][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:08:31,707][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:08:32,226][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:08:32,729][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:08:33,231][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:08:33,729][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:08:34,229][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:08:34,729][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:08:35,229][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:08:35,732][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:08:36,236][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:08:36,737][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:08:37,242][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:08:37,743][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:08:38,243][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:08:38,746][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:08:39,248][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:08:39,749][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:08:40,253][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:08:40,752][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:08:41,250][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:08:41,747][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:08:42,249][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:08:42,748][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:08:43,246][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:08:43,747][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:08:44,245][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:08:44,742][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:08:45,239][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:08:45,736][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:08:46,237][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:08:46,734][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:08:47,233][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:08:47,732][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:08:48,230][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:08:48,727][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:08:49,225][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:08:49,721][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:08:50,222][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:08:50,725][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:08:51,227][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:08:51,727][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:08:52,227][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:08:52,736][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:08:53,237][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:08:53,738][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:08:54,241][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:08:54,744][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:08:55,248][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10835 tokens.
[2025-11-13 00:08:56,034][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 00:08:56,823][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:08:56,825][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:08:56,827][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:08:57,734][__main__][INFO] - Iteration 135 took 52s (30.88% Gen, 67.39% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 35m 33s. Estimated total time: 43h 34m 49s. Time estimates for 10 more iterations: 8m 42s, 100 more iterations: 1h 27m 9s, 500 more iterations: 7h 15m 48s.
[2025-11-13 00:08:57,736][__main__][INFO] - Starting iteration 135.
[2025-11-13 00:08:58,214][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:08:58,214][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:09:05,993][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:09:14,261][__main__][INFO] - Number of regex retries in iteration 135: 1
[2025-11-13 00:09:14,262][__main__][INFO] - agents played in iteration 135 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:09:15,115][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:09:15,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:09:15,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:09:15,186][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:09:15,187][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:09:15,187][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:09:15,853][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:09:16,311][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:09:16,816][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:09:17,318][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:09:17,820][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:09:18,322][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:09:18,823][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:09:19,321][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:09:19,824][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:09:20,324][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:09:20,827][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:09:21,330][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:09:21,830][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:09:22,331][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:09:22,828][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:09:23,327][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:09:23,831][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:09:24,329][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:09:24,827][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:09:25,332][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:09:25,827][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:09:26,326][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:09:26,825][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:09:27,327][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:09:27,833][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:09:28,337][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:09:28,841][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:09:29,342][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:09:29,846][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:09:30,347][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:09:30,846][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:09:31,347][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:09:31,849][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:09:32,349][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:09:32,850][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:09:33,349][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:09:33,848][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:09:34,348][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:09:34,848][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:09:35,348][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:09:35,846][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:09:36,347][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:09:36,851][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:09:37,352][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:09:37,853][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:09:38,353][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:09:38,851][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:09:39,350][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:09:39,850][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:09:40,350][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:09:40,851][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:09:41,351][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:09:41,851][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:09:42,353][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:09:42,856][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:09:43,362][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:09:43,866][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:09:44,367][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:09:44,874][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:09:45,376][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:09:45,879][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:09:46,382][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:09:46,884][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:09:47,389][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:09:47,890][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10794 tokens.
[2025-11-13 00:09:48,557][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 00:09:49,316][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:09:49,318][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:09:49,320][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:09:50,263][__main__][INFO] - Iteration 136 took 52s (30.83% Gen, 67.35% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 22m 20s. Estimated total time: 43h 22m 28s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 44s, 500 more iterations: 7h 13m 44s.
[2025-11-13 00:09:50,265][__main__][INFO] - Starting iteration 136.
[2025-11-13 00:09:50,738][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:09:50,738][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:10:03,926][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:10:05,972][__main__][INFO] - Number of regex retries in iteration 136: 1
[2025-11-13 00:10:05,973][__main__][INFO] - agents played in iteration 136 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:10:06,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:10:06,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:10:06,825][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:10:06,847][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:10:06,847][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:10:06,848][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:10:07,508][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:10:07,963][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:10:08,469][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:10:08,967][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:10:09,467][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:10:09,965][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:10:10,463][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:10:10,965][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:10:11,464][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:10:11,963][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:10:12,463][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:10:12,961][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:10:13,461][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:10:13,960][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:10:14,461][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:10:14,961][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:10:15,460][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:10:15,959][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:10:16,459][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:10:16,958][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:10:17,458][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:10:17,960][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:10:18,461][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:10:18,960][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:10:19,458][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:10:19,958][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:10:20,467][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:10:20,969][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:10:21,470][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:10:21,972][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:10:22,471][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:10:22,971][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:10:23,473][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:10:23,973][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:10:24,472][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:10:24,970][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:10:25,473][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:10:25,972][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:10:26,473][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:10:26,970][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:10:27,468][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:10:27,966][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:10:28,464][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:10:28,962][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:10:29,463][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:10:29,960][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:10:30,462][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:10:30,963][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:10:31,463][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:10:31,965][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:10:32,467][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:10:32,971][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:10:33,473][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:10:33,973][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:10:34,474][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:10:34,975][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:10:35,481][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:10:35,985][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:10:36,489][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:10:36,991][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:10:37,493][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:10:37,996][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:10:38,496][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:10:38,998][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:10:39,498][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10808 tokens.
[2025-11-13 00:10:40,207][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 00:10:40,971][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:10:40,973][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:10:40,974][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:10:41,984][__main__][INFO] - Iteration 137 took 51s (29.73% Gen, 68.30% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 41m 20s. Estimated total time: 42h 42m 20s. Time estimates for 10 more iterations: 8m 32s, 100 more iterations: 1h 25m 24s, 500 more iterations: 7h 7m 3s.
[2025-11-13 00:10:41,986][__main__][INFO] - Starting iteration 137.
[2025-11-13 00:10:42,481][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:10:42,482][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:10:58,278][__main__][INFO] - Number of regex retries in iteration 137: 0
[2025-11-13 00:10:58,279][__main__][INFO] - agents played in iteration 137 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:10:59,068][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:10:59,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:10:59,121][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:10:59,144][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:10:59,144][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:10:59,145][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:10:59,794][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:11:00,252][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:11:00,755][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:11:01,259][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:11:01,758][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:11:02,260][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:11:02,759][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:11:03,256][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:11:03,763][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:11:04,263][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:11:04,764][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:11:05,264][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:11:05,763][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:11:06,265][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:11:06,764][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:11:07,263][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:11:07,767][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:11:08,265][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:11:08,769][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:11:09,265][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:11:09,761][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:11:10,262][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:11:10,762][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:11:11,262][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:11:11,767][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:11:12,267][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:11:12,772][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:11:13,274][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:11:13,777][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:11:14,282][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:11:14,785][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:11:15,286][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:11:15,789][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:11:16,290][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:11:16,791][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:11:17,295][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:11:17,795][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:11:18,299][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:11:18,800][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:11:19,301][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:11:19,800][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:11:20,301][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:11:20,802][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:11:21,305][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:11:21,804][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:11:22,306][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:11:22,807][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:11:23,308][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:11:23,810][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:11:24,311][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:11:24,816][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:11:25,317][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:11:25,817][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:11:26,317][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:11:26,819][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:11:27,318][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:11:27,820][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:11:28,320][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:11:28,824][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:11:29,323][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:11:29,826][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:11:30,325][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:11:30,825][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:11:31,325][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:11:31,827][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10779 tokens.
[2025-11-13 00:11:32,520][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 00:11:33,281][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:11:33,283][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:11:33,285][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:11:34,203][__main__][INFO] - Iteration 138 took 51s (30.54% Gen, 67.68% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 4m 14s. Estimated total time: 43h 6m 6s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 12s, 500 more iterations: 7h 11m 1s.
[2025-11-13 00:11:34,205][__main__][INFO] - Starting iteration 138.
[2025-11-13 00:11:34,704][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:11:34,704][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:11:51,948][__main__][INFO] - Number of regex retries in iteration 138: 0
[2025-11-13 00:11:51,949][__main__][INFO] - agents played in iteration 138 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:11:52,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:11:52,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:11:52,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:11:52,798][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:11:52,799][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:11:52,800][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:11:53,442][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:11:53,896][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:11:54,401][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:11:54,903][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:11:55,403][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:11:55,898][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:11:56,397][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:11:56,897][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:11:57,399][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:11:57,902][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:11:58,403][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:11:58,905][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:11:59,406][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:11:59,907][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:12:00,409][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:12:00,912][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:12:01,412][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:12:01,906][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:12:02,403][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:12:02,902][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:12:03,399][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:12:03,894][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:12:04,391][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:12:04,888][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:12:05,386][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:12:05,885][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:12:06,384][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:12:06,881][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:12:07,379][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:12:07,881][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:12:08,378][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:12:08,876][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:12:09,374][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:12:09,872][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:12:10,374][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:12:10,872][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:12:11,370][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:12:11,878][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:12:12,377][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:12:12,873][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:12:13,370][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:12:13,867][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:12:14,371][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:12:14,869][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:12:15,370][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:12:15,879][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:12:16,380][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:12:16,882][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:12:17,382][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:12:17,883][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:12:18,388][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:12:18,889][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:12:19,390][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:12:19,896][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:12:20,397][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:12:20,900][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:12:21,406][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:12:21,903][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:12:22,401][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:12:22,900][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:12:23,398][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:12:23,899][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:12:24,400][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:12:24,898][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:12:25,396][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10801 tokens.
[2025-11-13 00:12:26,054][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.51%, ΔTime: 00:00:32
[2025-11-13 00:12:26,830][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:12:26,831][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:12:26,833][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:12:27,757][__main__][INFO] - Iteration 139 took 53s (32.50% Gen, 65.75% Train). Generation: 17s, Training: 34s. Estimated remaining time: 42h 9m 54s. Estimated total time: 44h 12m 41s. Time estimates for 10 more iterations: 8m 50s, 100 more iterations: 1h 28m 25s, 500 more iterations: 7h 22m 6s.
[2025-11-13 00:12:27,759][__main__][INFO] - Starting iteration 139.
[2025-11-13 00:12:28,237][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:12:28,238][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:12:37,123][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:12:44,770][__main__][INFO] - Number of regex retries in iteration 139: 1
[2025-11-13 00:12:44,771][__main__][INFO] - agents played in iteration 139 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:12:45,552][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:12:45,575][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:12:45,597][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:12:45,620][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:12:45,621][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:12:45,621][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:12:46,260][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:12:46,714][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:12:47,221][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:12:47,723][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:12:48,223][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:12:48,724][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:12:49,222][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:12:49,722][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:12:50,221][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:12:50,720][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:12:51,222][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:12:51,725][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:12:52,221][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:12:52,719][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:12:53,218][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:12:53,716][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:12:54,213][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:12:54,710][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:12:55,208][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:12:55,705][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:12:56,202][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:12:56,702][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:12:57,203][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:12:57,703][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:12:58,205][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:12:58,702][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:12:59,203][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:12:59,702][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:13:00,201][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:13:00,700][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:13:01,201][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:13:01,701][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:13:02,199][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:13:02,698][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:13:03,197][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:13:03,695][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:13:04,193][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:13:04,691][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:13:05,187][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:13:05,692][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:13:06,189][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:13:06,691][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:13:07,189][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:13:07,694][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:13:08,201][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:13:08,702][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:13:09,199][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:13:09,703][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:13:10,205][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:13:10,707][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:13:11,209][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:13:11,710][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:13:12,213][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:13:12,713][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:13:13,213][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:13:13,717][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:13:14,220][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:13:14,720][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:13:15,222][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:13:15,721][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:13:16,220][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:13:16,719][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:13:17,218][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:13:17,716][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:13:18,215][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10793 tokens.
[2025-11-13 00:13:18,900][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:32
[2025-11-13 00:13:19,662][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:13:19,663][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:13:19,665][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:13:20,646][__main__][INFO] - Iteration 140 took 52s (31.55% Gen, 66.58% Train). Generation: 16s, Training: 34s. Estimated remaining time: 41h 36m 47s. Estimated total time: 43h 40m 26s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 20s, 500 more iterations: 7h 16m 44s.
[2025-11-13 00:13:20,648][__main__][INFO] - Starting iteration 140.
[2025-11-13 00:13:21,143][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 00:13:21,144][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:13:36,935][__main__][INFO] - Number of regex retries in iteration 140: 0
[2025-11-13 00:13:36,936][__main__][INFO] - agents played in iteration 140 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:13:37,734][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:13:37,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:13:37,796][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:13:37,819][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:13:37,819][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:13:37,820][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:13:38,460][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:13:38,915][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:13:39,419][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:13:39,921][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:13:40,423][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:13:40,921][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:13:41,419][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:13:41,918][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:13:42,417][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:13:42,917][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:13:43,418][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:13:43,916][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:13:44,414][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:13:44,909][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:13:45,402][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:13:45,899][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:13:46,399][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:13:46,897][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:13:47,399][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:13:47,898][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:13:48,396][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:13:48,902][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:13:49,406][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:13:49,904][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:13:50,409][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:13:50,914][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:13:51,410][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:13:51,910][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:13:52,412][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:13:52,913][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:13:53,413][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:13:53,912][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:13:54,413][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:13:54,909][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:13:55,411][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:13:55,910][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:13:56,408][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:13:56,914][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:13:57,415][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:13:57,917][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:13:58,416][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:13:58,917][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:13:59,421][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:13:59,922][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:14:00,424][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:14:00,924][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:14:01,425][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:14:01,928][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:14:02,429][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:14:02,928][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:14:03,429][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:14:03,928][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:14:04,426][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:14:04,924][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:14:05,422][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:14:05,924][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:14:06,425][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:14:06,923][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:14:07,421][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:14:07,919][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:14:08,422][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:14:08,920][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:14:09,418][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:14:09,920][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:14:10,419][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10781 tokens.
[2025-11-13 00:14:11,074][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 00:14:11,839][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:14:11,841][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:14:11,843][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:14:13,626][__main__][INFO] - Iteration 141 took 52s (30.09% Gen, 66.51% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 39m 37s. Estimated total time: 43h 44m 9s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 28s, 500 more iterations: 7h 17m 21s.
[2025-11-13 00:14:13,629][__main__][INFO] - Starting iteration 141.
[2025-11-13 00:14:14,102][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:14:14,103][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:14:30,045][__main__][INFO] - Number of regex retries in iteration 141: 0
[2025-11-13 00:14:30,046][__main__][INFO] - agents played in iteration 141 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:14:30,839][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:14:30,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:14:30,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:14:30,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:14:30,907][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:14:30,907][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:14:31,546][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:14:32,003][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:14:32,509][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:14:33,014][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:14:33,528][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:14:34,029][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:14:34,528][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:14:35,029][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:14:35,528][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:14:36,027][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:14:36,525][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:14:37,024][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:14:37,524][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:14:38,024][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:14:38,524][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:14:39,025][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:14:39,527][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:14:40,029][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:14:40,526][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:14:41,024][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:14:41,520][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:14:42,016][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:14:42,517][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:14:43,013][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:14:43,508][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:14:44,011][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:14:44,509][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:14:45,004][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:14:45,504][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:14:46,002][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:14:46,500][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:14:46,993][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:14:47,492][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:14:48,009][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:14:48,508][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:14:49,011][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:14:49,506][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:14:50,004][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:14:50,508][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:14:51,010][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:14:51,509][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:14:52,011][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:14:52,511][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:14:53,009][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:14:53,507][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:14:54,007][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:14:54,514][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:14:55,012][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:14:55,516][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:14:56,012][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:14:56,510][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:14:57,014][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:14:57,512][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:14:58,009][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:14:58,511][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:14:59,009][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:14:59,506][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:15:00,001][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:15:00,495][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:15:00,997][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:15:01,491][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:15:01,987][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:15:02,485][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:15:02,980][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:15:03,475][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10654 tokens.
[2025-11-13 00:15:04,174][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 00:15:04,951][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:15:04,953][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:15:04,954][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:15:05,867][__main__][INFO] - Iteration 142 took 51s (30.80% Gen, 67.44% Train). Generation: 15s, Training: 34s. Estimated remaining time: 41h 2m 51s. Estimated total time: 43h 8m 15s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 16s, 500 more iterations: 7h 11m 22s.
[2025-11-13 00:15:05,869][__main__][INFO] - Starting iteration 142.
[2025-11-13 00:15:06,345][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:15:06,345][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:15:09,398][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:15:22,201][__main__][INFO] - Number of regex retries in iteration 142: 1
[2025-11-13 00:15:22,202][__main__][INFO] - agents played in iteration 142 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:15:23,073][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:15:23,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:15:23,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:15:23,145][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:15:23,146][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:15:23,146][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:15:23,757][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:15:24,209][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:15:24,715][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:15:25,216][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:15:25,716][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:15:26,216][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:15:26,715][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:15:27,231][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:15:27,729][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:15:28,225][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:15:28,726][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:15:29,225][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:15:29,726][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:15:30,227][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:15:30,730][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:15:31,233][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:15:31,735][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:15:32,238][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:15:32,740][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:15:33,240][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:15:33,747][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:15:34,253][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:15:34,752][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:15:35,253][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:15:35,754][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:15:36,251][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:15:36,748][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:15:37,249][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:15:37,759][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:15:38,264][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:15:38,764][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:15:39,270][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:15:39,770][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:15:40,271][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:15:40,772][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:15:41,273][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:15:41,779][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:15:42,281][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:15:42,783][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:15:43,284][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:15:43,785][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:15:44,287][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:15:44,791][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:15:45,291][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:15:45,806][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:15:46,307][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:15:46,818][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:15:47,318][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:15:47,819][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:15:48,323][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:15:48,823][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:15:49,320][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:15:49,819][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:15:50,320][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:15:50,830][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:15:51,328][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:15:51,827][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:15:52,325][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:15:52,824][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:15:53,332][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:15:53,834][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:15:54,335][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:15:54,838][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:15:55,336][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:15:55,837][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10694 tokens.
[2025-11-13 00:15:56,516][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 00:15:57,319][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:15:57,320][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:15:57,322][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:15:58,368][__main__][INFO] - Iteration 143 took 52s (30.48% Gen, 67.51% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 14m 55s. Estimated total time: 43h 21m 11s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 42s, 500 more iterations: 7h 13m 31s.
[2025-11-13 00:15:58,370][__main__][INFO] - Starting iteration 143.
[2025-11-13 00:15:58,844][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:15:58,845][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:16:15,725][__main__][INFO] - Number of regex retries in iteration 143: 0
[2025-11-13 00:16:15,726][__main__][INFO] - agents played in iteration 143 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:16:16,565][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:16:16,589][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:16:16,614][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:16:16,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:16:16,637][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:16:16,637][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:16:17,253][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:16:17,711][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:16:18,230][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:16:18,730][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:16:19,232][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:16:19,732][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:16:20,233][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:16:20,734][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:16:21,232][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:16:21,733][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:16:22,244][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:16:22,745][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:16:23,247][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:16:23,748][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:16:24,248][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:16:24,752][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:16:25,249][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:16:25,748][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:16:26,253][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:16:26,752][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:16:27,257][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:16:27,759][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:16:28,265][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:16:28,768][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:16:29,269][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:16:29,770][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:16:30,273][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:16:30,773][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:16:31,273][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:16:31,774][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:16:32,275][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:16:32,780][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:16:33,281][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:16:33,783][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:16:34,287][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:16:34,790][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:16:35,299][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:16:35,801][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:16:36,305][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:16:36,815][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:16:37,317][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:16:37,827][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:16:38,332][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:16:38,833][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:16:39,336][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:16:39,836][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:16:40,339][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:16:40,847][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:16:41,350][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:16:41,852][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:16:42,351][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:16:42,851][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:16:43,347][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:16:43,846][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:16:44,345][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:16:44,845][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:16:45,344][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:16:45,839][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:16:46,332][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:16:46,825][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:16:47,338][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:16:47,841][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:16:48,347][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:16:48,845][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:16:49,346][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10775 tokens.
[2025-11-13 00:16:50,039][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 00:16:50,790][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:16:50,792][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:16:50,794][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:16:51,675][__main__][INFO] - Iteration 144 took 52s (31.95% Gen, 66.38% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 54m 23s. Estimated total time: 44h 1m 33s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 3s, 500 more iterations: 7h 20m 15s.
[2025-11-13 00:16:51,677][__main__][INFO] - Starting iteration 144.
[2025-11-13 00:16:52,163][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:16:52,163][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:17:07,734][__main__][INFO] - Number of regex retries in iteration 144: 0
[2025-11-13 00:17:07,735][__main__][INFO] - agents played in iteration 144 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:17:08,549][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:17:08,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:17:08,598][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:17:08,620][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:17:08,621][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:17:08,621][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:17:09,229][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:17:09,691][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:17:10,191][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:17:10,689][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:17:11,189][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:17:11,687][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:17:12,185][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:17:12,687][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:17:13,187][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:17:13,691][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:17:14,195][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:17:14,697][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:17:15,202][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:17:15,704][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:17:16,219][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:17:16,723][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:17:17,226][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:17:17,731][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:17:18,233][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:17:18,747][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:17:19,248][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:17:19,751][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:17:20,249][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:17:20,751][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:17:21,252][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:17:21,754][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:17:22,255][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:17:22,762][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:17:23,263][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:17:23,764][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:17:24,264][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:17:24,766][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:17:25,269][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:17:25,770][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:17:26,271][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:17:26,774][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:17:27,274][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:17:27,773][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:17:28,273][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:17:28,774][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:17:29,278][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:17:29,780][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:17:30,282][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:17:30,788][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:17:31,291][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:17:31,794][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:17:32,298][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:17:32,803][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:17:33,320][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:17:33,824][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:17:34,336][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:17:34,834][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:17:35,334][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:17:35,834][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:17:36,331][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:17:36,829][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:17:37,327][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:17:37,829][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:17:38,347][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:17:38,848][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:17:39,351][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:17:39,857][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:17:40,359][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:17:40,862][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:17:41,362][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10791 tokens.
[2025-11-13 00:17:42,040][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.29%, Current % of VRAM taken: 60.55%, Block Peak % of device VRAM: 62.52%, ΔTime: 00:00:32
[2025-11-13 00:17:42,793][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:17:42,795][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:17:42,796][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:17:43,747][__main__][INFO] - Iteration 145 took 51s (30.18% Gen, 67.97% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 51m 13s. Estimated total time: 42h 59m 15s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 58s, 500 more iterations: 7h 9m 52s.
[2025-11-13 00:17:43,750][__main__][INFO] - Starting iteration 145.
[2025-11-13 00:17:44,230][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:17:44,231][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:18:01,125][__main__][INFO] - Number of regex retries in iteration 145: 0
[2025-11-13 00:18:01,126][__main__][INFO] - agents played in iteration 145 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:18:01,938][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:18:01,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:18:01,982][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:18:02,004][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:18:02,004][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:18:02,005][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:18:02,616][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:18:03,072][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:18:03,580][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:18:04,078][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:18:04,575][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:18:05,071][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:18:05,564][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:18:06,065][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:18:06,566][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:18:07,072][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:18:07,575][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:18:08,079][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:18:08,582][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:18:09,086][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:18:09,587][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:18:10,095][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:18:10,597][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:18:11,101][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:18:11,612][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:18:12,113][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:18:12,612][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:18:13,112][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:18:13,611][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:18:14,112][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:18:14,610][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:18:15,109][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:18:15,607][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:18:16,106][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:18:16,611][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:18:17,108][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:18:17,604][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:18:18,105][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:18:18,604][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:18:19,103][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:18:19,603][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:18:20,102][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:18:20,604][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:18:21,108][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:18:21,609][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:18:22,125][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:18:22,624][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:18:23,157][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:18:23,654][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:18:24,160][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:18:24,667][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:18:25,169][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:18:25,666][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:18:26,164][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:18:26,662][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:18:27,163][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:18:27,660][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:18:28,158][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:18:28,655][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:18:29,148][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:18:29,649][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:18:30,146][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:18:30,649][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:18:31,150][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:18:31,652][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:18:32,156][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:18:32,654][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:18:33,156][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:18:33,664][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:18:34,166][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:18:34,666][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10806 tokens.
[2025-11-13 00:18:35,336][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.40%, ΔTime: 00:00:32
[2025-11-13 00:18:36,089][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:18:36,090][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:18:36,092][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:18:37,032][__main__][INFO] - Iteration 146 took 52s (32.00% Gen, 66.22% Train). Generation: 16s, Training: 34s. Estimated remaining time: 41h 51m 11s. Estimated total time: 44h 0m 6s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 0s, 500 more iterations: 7h 20m 1s.
[2025-11-13 00:18:37,034][__main__][INFO] - Starting iteration 146.
[2025-11-13 00:18:37,537][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:18:37,538][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:18:47,238][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:18:55,384][__main__][INFO] - Number of regex retries in iteration 146: 1
[2025-11-13 00:18:55,384][__main__][INFO] - agents played in iteration 146 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:18:56,420][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:18:56,447][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:18:56,473][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:18:56,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:18:56,496][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:18:56,497][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:18:57,137][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:18:57,606][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:18:58,117][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:18:58,628][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:18:59,128][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:18:59,629][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:19:00,146][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:19:00,647][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:19:01,148][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:19:01,645][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:19:02,146][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:19:02,661][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:19:03,162][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:19:03,660][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:19:04,163][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:19:04,666][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:19:05,168][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:19:05,667][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:19:06,166][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:19:06,666][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:19:07,166][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:19:07,667][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:19:08,165][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:19:08,663][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:19:09,167][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:19:09,668][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:19:10,169][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:19:10,670][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:19:11,170][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:19:11,670][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:19:12,169][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:19:12,670][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:19:13,176][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:19:13,678][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:19:14,173][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:19:14,671][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:19:15,171][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:19:15,676][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:19:16,176][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:19:16,677][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:19:17,175][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:19:17,670][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:19:18,191][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:19:18,686][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:19:19,184][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:19:19,680][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:19:20,180][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:19:20,678][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:19:21,175][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:19:21,671][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:19:22,172][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:19:22,673][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:19:23,172][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:19:23,673][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:19:24,170][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:19:24,667][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:19:25,168][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:19:25,668][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:19:26,170][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:19:26,669][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:19:27,171][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:19:27,674][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:19:28,172][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:19:28,670][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:19:29,174][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10777 tokens.
[2025-11-13 00:19:29,823][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 00:19:30,594][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:19:30,596][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:19:30,598][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:19:31,562][__main__][INFO] - Iteration 147 took 54s (33.03% Gen, 65.18% Train). Generation: 17s, Training: 35s. Estimated remaining time: 42h 51m 26s. Estimated total time: 45h 1m 16s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 2s, 500 more iterations: 7h 30m 12s.
[2025-11-13 00:19:31,564][__main__][INFO] - Starting iteration 147.
[2025-11-13 00:19:32,034][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:19:32,035][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:19:47,011][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:19:47,886][__main__][INFO] - Number of regex retries in iteration 147: 1
[2025-11-13 00:19:47,886][__main__][INFO] - agents played in iteration 147 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:19:48,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:19:48,781][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:19:48,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:19:48,828][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:19:48,828][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:19:48,829][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:19:49,527][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:19:49,992][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:19:50,497][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:19:51,000][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:19:51,503][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:19:52,004][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:19:52,508][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:19:53,008][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:19:53,509][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:19:54,019][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:19:54,521][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:19:55,023][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:19:55,527][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:19:56,026][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:19:56,545][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:19:57,045][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:19:57,541][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:19:58,041][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:19:58,539][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:19:59,048][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:19:59,548][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:20:00,044][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:20:00,550][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:20:01,049][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:20:01,547][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:20:02,047][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:20:02,546][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:20:03,047][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:20:03,546][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:20:04,046][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:20:04,562][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:20:05,062][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:20:05,563][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:20:06,072][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:20:06,572][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:20:07,074][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:20:07,576][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:20:08,077][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:20:08,577][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:20:09,078][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:20:09,577][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:20:10,076][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:20:10,573][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:20:11,071][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:20:11,566][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:20:12,063][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:20:12,564][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:20:13,062][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:20:13,564][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:20:14,062][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:20:14,562][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:20:15,065][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:20:15,566][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:20:16,064][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:20:16,564][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:20:17,064][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:20:17,566][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:20:18,068][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:20:18,567][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:20:19,070][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:20:19,568][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:20:20,068][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:20:20,569][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:20:21,072][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:20:21,572][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10726 tokens.
[2025-11-13 00:20:22,247][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 00:20:23,050][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:20:23,051][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:20:23,053][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:20:23,962][__main__][INFO] - Iteration 148 took 51s (30.53% Gen, 67.72% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 5m 44s. Estimated total time: 43h 16m 26s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 32s, 500 more iterations: 7h 12m 44s.
[2025-11-13 00:20:23,967][__main__][INFO] - Starting iteration 148.
[2025-11-13 00:20:24,462][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:20:24,462][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:20:37,956][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:20:41,296][__main__][INFO] - Number of regex retries in iteration 148: 1
[2025-11-13 00:20:41,296][__main__][INFO] - agents played in iteration 148 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:20:42,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:20:42,120][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:20:42,146][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:20:42,170][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:20:42,171][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:20:42,172][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:20:42,854][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:20:43,325][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:20:43,836][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:20:44,337][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:20:44,837][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:20:45,340][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:20:45,842][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:20:46,343][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:20:46,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:20:47,342][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:20:47,857][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:20:48,356][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:20:48,854][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:20:49,354][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:20:49,856][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:20:50,355][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:20:50,854][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:20:51,352][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:20:51,866][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:20:52,363][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:20:52,863][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:20:53,360][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:20:53,858][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:20:54,366][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:20:54,864][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:20:55,367][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:20:55,884][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:20:56,386][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:20:56,888][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:20:57,390][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:20:57,892][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:20:58,396][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:20:58,899][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:20:59,399][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:20:59,899][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:21:00,400][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:21:00,903][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:21:01,398][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:21:01,896][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:21:02,395][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:21:02,891][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:21:03,387][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:21:03,883][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:21:04,381][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:21:04,886][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:21:05,386][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:21:05,889][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:21:06,391][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:21:06,890][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:21:07,393][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:21:07,891][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:21:08,388][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:21:08,894][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:21:09,393][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:21:09,892][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:21:10,395][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:21:10,896][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:21:11,398][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:21:11,899][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:21:12,398][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:21:12,919][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:21:13,418][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:21:13,915][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:21:14,415][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:21:14,915][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10698 tokens.
[2025-11-13 00:21:15,589][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 00:21:16,357][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:21:16,359][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:21:16,361][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:21:17,326][__main__][INFO] - Iteration 149 took 52s (31.84% Gen, 66.33% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 51m 39s. Estimated total time: 44h 3m 14s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 6s, 500 more iterations: 7h 20m 32s.
[2025-11-13 00:21:17,329][__main__][INFO] - Starting iteration 149.
[2025-11-13 00:21:17,818][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:21:17,819][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:21:21,506][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:21:34,636][__main__][INFO] - Number of regex retries in iteration 149: 1
[2025-11-13 00:21:34,637][__main__][INFO] - agents played in iteration 149 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:21:35,507][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:21:35,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:21:35,552][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:21:35,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:21:35,574][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:21:35,575][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:21:36,262][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:21:36,722][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:21:37,229][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:21:37,734][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:21:38,236][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:21:38,736][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:21:39,240][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:21:39,741][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:21:40,237][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:21:40,736][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:21:41,239][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:21:41,737][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:21:42,238][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:21:42,735][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:21:43,236][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:21:43,735][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:21:44,232][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:21:44,729][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:21:45,226][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:21:45,725][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:21:46,224][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:21:46,722][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:21:47,222][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:21:47,720][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:21:48,223][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:21:48,719][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:21:49,221][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:21:49,724][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:21:50,224][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:21:50,725][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:21:51,225][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:21:51,728][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:21:52,229][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:21:52,725][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:21:53,222][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:21:53,721][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:21:54,219][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:21:54,718][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:21:55,214][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:21:55,712][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:21:56,210][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:21:56,708][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:21:57,207][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:21:57,707][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:21:58,204][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:21:58,706][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:21:59,208][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:21:59,708][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:22:00,211][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:22:00,712][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:22:01,214][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:22:01,721][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:22:02,222][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:22:02,724][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:22:03,224][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:22:03,724][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:22:04,226][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:22:04,723][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:22:05,221][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:22:05,714][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:22:06,216][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:22:06,719][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:22:07,216][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:22:07,711][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:22:08,208][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10723 tokens.
[2025-11-13 00:22:08,866][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.36%, ΔTime: 00:00:32
[2025-11-13 00:22:09,652][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:22:09,653][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:22:09,655][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:22:10,635][__main__][INFO] - Iteration 150 took 52s (31.84% Gen, 66.30% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 48m 23s. Estimated total time: 44h 0m 52s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 1s, 500 more iterations: 7h 20m 8s.
[2025-11-13 00:22:10,637][__main__][INFO] - Starting iteration 150.
[2025-11-13 00:22:11,108][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 00:22:11,108][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:22:26,077][__main__][INFO] - Number of regex retries in iteration 150: 0
[2025-11-13 00:22:26,078][__main__][INFO] - agents played in iteration 150 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:22:26,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:22:26,907][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:22:26,930][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:22:26,952][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:22:26,953][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:22:26,953][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:22:27,637][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:22:28,095][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:22:28,602][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:22:29,104][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:22:29,604][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:22:30,115][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:22:30,617][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:22:31,117][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:22:31,616][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:22:32,116][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:22:32,614][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:22:33,114][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:22:33,616][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:22:34,119][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:22:34,620][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:22:35,122][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:22:35,623][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:22:36,123][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:22:36,626][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:22:37,127][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:22:37,627][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:22:38,127][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:22:38,627][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:22:39,127][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:22:39,627][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:22:40,131][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:22:40,649][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:22:41,149][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:22:41,649][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:22:42,153][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:22:42,657][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:22:43,160][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:22:43,662][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:22:44,161][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:22:44,665][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:22:45,164][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:22:45,659][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:22:46,167][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:22:46,665][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:22:47,163][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:22:47,662][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:22:48,161][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:22:48,670][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:22:49,165][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:22:49,668][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:22:50,180][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:22:50,681][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:22:51,193][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:22:51,697][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:22:52,196][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:22:52,699][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:22:53,198][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:22:53,698][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:22:54,198][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:22:54,696][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:22:55,210][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:22:55,710][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:22:56,208][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:22:56,711][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:22:57,213][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:22:57,712][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:22:58,214][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:22:58,714][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:22:59,219][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:22:59,719][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10822 tokens.
[2025-11-13 00:23:00,375][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.28%, Current % of VRAM taken: 60.54%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 00:23:01,142][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:23:01,144][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:23:01,146][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:23:03,099][__main__][INFO] - Iteration 151 took 51s (28.79% Gen, 67.45% Train). Generation: 14s, Training: 35s. Estimated remaining time: 41h 6m 15s. Estimated total time: 43h 19m 36s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 39s, 500 more iterations: 7h 13m 16s.
[2025-11-13 00:23:03,101][__main__][INFO] - Starting iteration 151.
[2025-11-13 00:23:03,584][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:23:03,585][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:23:10,653][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:23:20,732][__main__][INFO] - Number of regex retries in iteration 151: 1
[2025-11-13 00:23:20,732][__main__][INFO] - agents played in iteration 151 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:23:21,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:23:21,613][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:23:21,635][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:23:21,657][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:23:21,658][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:23:21,658][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:23:22,308][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:23:22,767][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:23:23,272][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:23:23,773][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:23:24,286][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:23:24,784][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:23:25,284][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:23:25,789][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:23:26,285][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:23:26,784][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:23:27,285][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:23:27,806][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:23:28,323][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:23:28,823][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:23:29,326][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:23:29,826][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:23:30,325][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:23:30,828][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:23:31,327][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:23:31,830][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:23:32,335][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:23:32,837][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:23:33,341][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:23:33,842][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:23:34,342][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:23:34,852][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:23:35,353][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:23:35,855][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:23:36,354][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:23:36,856][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:23:37,360][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:23:37,862][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:23:38,361][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:23:38,859][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:23:39,355][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:23:39,849][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:23:40,343][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:23:40,838][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:23:41,334][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:23:41,832][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:23:42,331][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:23:42,833][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:23:43,334][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:23:43,837][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:23:44,335][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:23:44,837][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:23:45,338][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:23:45,839][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:23:46,339][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:23:46,840][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:23:47,340][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:23:47,839][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:23:48,340][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:23:48,838][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:23:49,339][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:23:49,835][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:23:50,333][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:23:50,829][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:23:51,324][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:23:51,820][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:23:52,316][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:23:52,811][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:23:53,320][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:23:53,816][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:23:54,311][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10758 tokens.
[2025-11-13 00:23:54,966][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.50%, ΔTime: 00:00:32
[2025-11-13 00:23:55,745][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:23:55,747][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:23:55,749][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:23:56,685][__main__][INFO] - Iteration 152 took 53s (32.29% Gen, 65.94% Train). Generation: 17s, Training: 35s. Estimated remaining time: 42h 0m 46s. Estimated total time: 44h 15m 1s. Time estimates for 10 more iterations: 8m 51s, 100 more iterations: 1h 28m 30s, 500 more iterations: 7h 22m 30s.
[2025-11-13 00:23:56,687][__main__][INFO] - Starting iteration 152.
[2025-11-13 00:23:57,193][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:23:57,193][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:24:12,937][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:24:13,849][__main__][INFO] - Number of regex retries in iteration 152: 1
[2025-11-13 00:24:13,850][__main__][INFO] - agents played in iteration 152 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:24:14,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:24:14,711][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:24:14,738][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:24:14,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:24:14,763][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:24:14,764][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:24:15,398][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:24:15,859][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:24:16,366][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:24:16,866][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:24:17,367][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:24:17,868][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:24:18,371][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:24:18,872][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:24:19,368][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:24:19,870][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:24:20,370][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:24:20,874][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:24:21,374][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:24:21,874][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:24:22,375][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:24:22,874][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:24:23,372][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:24:23,871][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:24:24,373][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:24:24,891][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:24:25,394][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:24:25,896][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:24:26,411][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:24:26,910][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:24:27,408][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:24:27,905][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:24:28,402][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:24:28,918][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:24:29,416][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:24:29,913][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:24:30,409][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:24:30,906][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:24:31,411][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:24:31,910][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:24:32,406][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:24:32,922][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:24:33,424][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:24:33,927][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:24:34,430][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:24:34,932][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:24:35,436][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:24:35,936][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:24:36,439][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:24:36,943][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:24:37,445][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:24:37,947][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:24:38,450][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:24:38,951][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:24:39,454][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:24:39,952][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:24:40,445][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:24:40,943][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:24:41,442][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:24:41,942][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:24:42,439][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:24:42,939][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:24:43,439][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:24:43,939][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:24:44,437][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:24:44,935][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:24:45,432][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:24:45,932][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:24:46,428][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:24:46,925][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:24:47,424][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10763 tokens.
[2025-11-13 00:24:48,079][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.38%, ΔTime: 00:00:32
[2025-11-13 00:24:49,035][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:24:49,036][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:24:49,055][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:24:50,321][__main__][INFO] - Iteration 153 took 53s (31.35% Gen, 66.27% Train). Generation: 16s, Training: 35s. Estimated remaining time: 42h 1m 16s. Estimated total time: 44h 16m 24s. Time estimates for 10 more iterations: 8m 51s, 100 more iterations: 1h 28m 32s, 500 more iterations: 7h 22m 44s.
[2025-11-13 00:24:50,323][__main__][INFO] - Starting iteration 153.
[2025-11-13 00:24:50,790][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:24:50,790][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:25:06,639][__main__][INFO] - Number of regex retries in iteration 153: 0
[2025-11-13 00:25:06,640][__main__][INFO] - agents played in iteration 153 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:25:07,445][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:25:07,470][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:25:07,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:25:07,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:25:07,517][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:25:07,518][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:25:08,179][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:25:08,637][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:25:09,144][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:25:09,645][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:25:10,146][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:25:10,651][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:25:11,153][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:25:11,654][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:25:12,158][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:25:12,660][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:25:13,162][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:25:13,665][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:25:14,168][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:25:14,676][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:25:15,179][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:25:15,683][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:25:16,190][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:25:16,692][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:25:17,194][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:25:17,710][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:25:18,213][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:25:18,726][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:25:19,226][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:25:19,726][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:25:20,230][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:25:20,729][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:25:21,230][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:25:21,729][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:25:22,227][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:25:22,734][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:25:23,232][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:25:23,730][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:25:24,227][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:25:24,726][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:25:25,227][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:25:25,725][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:25:26,224][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:25:26,735][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:25:27,239][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:25:27,742][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:25:28,244][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:25:28,746][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:25:29,248][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:25:29,749][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:25:30,250][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:25:30,753][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:25:31,251][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:25:31,752][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:25:32,255][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:25:32,755][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:25:33,253][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:25:33,756][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:25:34,255][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:25:34,755][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:25:35,252][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:25:35,750][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:25:36,251][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:25:36,752][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:25:37,249][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:25:37,750][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:25:38,248][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:25:38,747][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:25:39,244][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:25:39,746][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:25:40,241][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10762 tokens.
[2025-11-13 00:25:40,895][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 00:25:41,682][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:25:41,684][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:25:41,686][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:25:42,697][__main__][INFO] - Iteration 154 took 51s (30.53% Gen, 67.52% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 59m 22s. Estimated total time: 43h 15m 23s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 30s, 500 more iterations: 7h 12m 33s.
[2025-11-13 00:25:42,699][__main__][INFO] - Starting iteration 154.
[2025-11-13 00:25:43,185][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:25:43,186][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:26:00,005][__main__][INFO] - Number of regex retries in iteration 154: 0
[2025-11-13 00:26:00,006][__main__][INFO] - agents played in iteration 154 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:26:00,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:26:00,834][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:26:00,860][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:26:00,882][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:26:00,883][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:26:00,884][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:26:01,512][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:26:01,970][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:26:02,477][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:26:02,977][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:26:03,485][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:26:03,990][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:26:04,495][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:26:05,005][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:26:05,506][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:26:06,010][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:26:06,512][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:26:07,013][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:26:07,518][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:26:08,021][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:26:08,521][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:26:09,020][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:26:09,517][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:26:10,030][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:26:10,529][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:26:11,026][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:26:11,527][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:26:12,025][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:26:12,534][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:26:13,032][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:26:13,530][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:26:14,028][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:26:14,525][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:26:15,021][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:26:15,521][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:26:16,019][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:26:16,539][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:26:17,035][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:26:17,532][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:26:18,029][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:26:18,528][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:26:19,039][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:26:19,539][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:26:20,040][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:26:20,545][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:26:21,043][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:26:21,547][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:26:22,047][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:26:22,546][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:26:23,049][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:26:23,549][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:26:24,049][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:26:24,549][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:26:25,047][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:26:25,552][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:26:26,048][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:26:26,550][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:26:27,047][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:26:27,546][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:26:28,046][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:26:28,544][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:26:29,044][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:26:29,551][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:26:30,049][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:26:30,550][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:26:31,047][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:26:31,548][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:26:32,047][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:26:32,546][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:26:33,045][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:26:33,553][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10691 tokens.
[2025-11-13 00:26:34,203][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 00:26:34,984][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:26:34,986][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:26:34,987][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:26:35,915][__main__][INFO] - Iteration 155 took 52s (31.90% Gen, 66.34% Train). Generation: 16s, Training: 34s. Estimated remaining time: 41h 39m 36s. Estimated total time: 43h 56m 31s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 53s, 500 more iterations: 7h 19m 25s.
[2025-11-13 00:26:35,917][__main__][INFO] - Starting iteration 155.
[2025-11-13 00:26:36,427][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:26:36,428][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:26:53,174][__main__][INFO] - Number of regex retries in iteration 155: 0
[2025-11-13 00:26:53,174][__main__][INFO] - agents played in iteration 155 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:26:54,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:26:54,068][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:26:54,095][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:26:54,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:26:54,119][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:26:54,121][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:26:54,775][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:26:55,234][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:26:55,738][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:26:56,241][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:26:56,741][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:26:57,241][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:26:57,745][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:26:58,250][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:26:58,755][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:26:59,251][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:26:59,748][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:27:00,253][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:27:00,750][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:27:01,250][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:27:01,747][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:27:02,243][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:27:02,750][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:27:03,244][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:27:03,743][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:27:04,243][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:27:04,738][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:27:05,236][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:27:05,734][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:27:06,231][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:27:06,744][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:27:07,245][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:27:07,743][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:27:08,243][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:27:08,741][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:27:09,245][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:27:09,740][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:27:10,242][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:27:10,753][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:27:11,253][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:27:11,759][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:27:12,259][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:27:12,759][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:27:13,265][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:27:13,764][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:27:14,264][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:27:14,768][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:27:15,269][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:27:15,768][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:27:16,267][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:27:16,765][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:27:17,267][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:27:17,763][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:27:18,262][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:27:18,762][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:27:19,259][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:27:19,754][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:27:20,249][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:27:20,745][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:27:21,250][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:27:21,744][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:27:22,244][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:27:22,740][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:27:23,239][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:27:23,743][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:27:24,237][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:27:24,736][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:27:25,238][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:27:25,734][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:27:26,230][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:27:26,730][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10708 tokens.
[2025-11-13 00:27:27,419][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 00:27:28,210][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:27:28,212][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:27:28,213][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:27:29,244][__main__][INFO] - Iteration 156 took 52s (31.71% Gen, 66.34% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 43m 4s. Estimated total time: 44h 0m 52s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 1s, 500 more iterations: 7h 20m 8s.
[2025-11-13 00:27:29,246][__main__][INFO] - Starting iteration 156.
[2025-11-13 00:27:29,714][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:27:29,715][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:27:33,099][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:27:33,102][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:27:45,851][__main__][INFO] - Number of regex retries in iteration 156: 2
[2025-11-13 00:27:45,851][__main__][INFO] - agents played in iteration 156 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:27:46,712][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:27:46,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:27:46,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:27:46,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:27:46,779][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:27:46,781][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:27:47,462][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:27:47,922][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:27:48,430][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:27:48,938][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:27:49,441][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:27:49,953][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:27:50,453][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:27:50,956][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:27:51,463][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:27:51,964][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:27:52,464][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:27:52,962][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:27:53,457][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:27:53,962][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:27:54,459][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:27:54,953][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:27:55,461][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:27:55,959][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:27:56,460][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:27:56,952][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:27:57,450][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:27:57,960][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:27:58,460][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:27:58,957][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:27:59,465][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:27:59,964][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:28:00,468][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:28:00,972][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:28:01,474][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:28:01,976][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:28:02,482][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:28:02,982][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:28:03,486][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:28:03,986][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:28:04,487][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:28:04,987][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:28:05,487][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:28:05,990][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:28:06,492][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:28:06,992][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:28:07,491][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:28:07,989][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:28:08,491][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:28:08,990][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:28:09,490][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:28:09,991][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:28:10,492][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:28:10,992][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:28:11,492][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:28:11,988][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:28:12,487][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:28:12,985][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:28:13,483][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:28:13,985][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:28:14,484][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:28:14,984][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:28:15,483][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:28:15,983][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:28:16,485][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:28:16,983][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:28:17,483][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:28:17,986][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:28:18,487][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:28:18,986][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:28:19,485][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10694 tokens.
[2025-11-13 00:28:20,182][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.45%, ΔTime: 00:00:32
[2025-11-13 00:28:20,956][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:28:20,957][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:28:20,959][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:28:21,939][__main__][INFO] - Iteration 157 took 52s (30.90% Gen, 67.22% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 12m 34s. Estimated total time: 43h 31m 14s. Time estimates for 10 more iterations: 8m 42s, 100 more iterations: 1h 27m 2s, 500 more iterations: 7h 15m 12s.
[2025-11-13 00:28:21,941][__main__][INFO] - Starting iteration 157.
[2025-11-13 00:28:22,450][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:28:22,450][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:28:25,806][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:28:37,945][__main__][INFO] - Number of regex retries in iteration 157: 1
[2025-11-13 00:28:37,946][__main__][INFO] - agents played in iteration 157 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:28:38,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:28:38,814][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:28:38,839][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:28:38,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:28:38,862][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:28:38,863][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:28:39,516][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:28:39,979][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:28:40,484][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:28:40,988][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:28:41,489][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:28:41,991][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:28:42,492][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:28:42,989][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:28:43,487][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:28:43,987][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:28:44,484][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:28:44,985][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:28:45,483][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:28:45,982][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:28:46,482][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:28:46,980][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:28:47,476][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:28:47,978][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:28:48,477][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:28:48,975][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:28:49,472][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:28:49,970][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:28:50,472][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:28:50,970][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:28:51,470][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:28:51,971][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:28:52,467][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:28:52,962][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:28:53,460][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:28:53,961][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:28:54,467][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:28:54,967][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:28:55,468][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:28:55,972][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:28:56,470][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:28:56,973][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:28:57,473][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:28:57,973][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:28:58,481][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:28:58,980][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:28:59,482][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:28:59,980][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:29:00,479][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:29:00,982][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:29:01,482][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:29:01,977][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:29:02,477][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:29:02,974][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:29:03,473][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:29:03,972][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:29:04,468][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:29:04,974][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:29:05,471][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:29:05,973][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:29:06,470][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:29:06,970][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:29:07,472][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:29:07,969][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:29:08,468][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:29:08,968][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:29:09,470][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:29:09,970][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:29:10,467][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:29:10,967][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:29:11,472][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10767 tokens.
[2025-11-13 00:29:12,182][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 00:29:12,957][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:29:12,959][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:29:12,960][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:29:14,044][__main__][INFO] - Iteration 158 took 51s (30.03% Gen, 67.86% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 40m 12s. Estimated total time: 42h 59m 45s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 59s, 500 more iterations: 7h 9m 57s.
[2025-11-13 00:29:14,046][__main__][INFO] - Starting iteration 158.
[2025-11-13 00:29:14,549][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:29:14,550][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:29:31,851][__main__][INFO] - Number of regex retries in iteration 158: 0
[2025-11-13 00:29:31,852][__main__][INFO] - agents played in iteration 158 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:29:32,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:29:32,764][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:29:32,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:29:32,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:29:32,814][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:29:32,814][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:29:33,443][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:29:33,902][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:29:34,405][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:29:34,906][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:29:35,409][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:29:35,909][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:29:36,409][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:29:36,909][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:29:37,408][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:29:37,907][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:29:38,406][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:29:38,904][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:29:39,405][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:29:39,905][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:29:40,404][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:29:40,903][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:29:41,402][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:29:41,903][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:29:42,403][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:29:42,902][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:29:43,399][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:29:43,899][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:29:44,398][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:29:44,899][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:29:45,401][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:29:45,904][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:29:46,410][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:29:46,914][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:29:47,420][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:29:47,926][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:29:48,426][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:29:48,930][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:29:49,431][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:29:49,945][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:29:50,447][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:29:50,952][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:29:51,449][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:29:51,949][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:29:52,446][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:29:52,945][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:29:53,441][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:29:53,951][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:29:54,454][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:29:54,955][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:29:55,456][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:29:55,958][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:29:56,476][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:29:56,975][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:29:57,479][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:29:57,974][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:29:58,474][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:29:58,985][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:29:59,483][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:29:59,980][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:30:00,488][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:30:00,985][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:30:01,484][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:30:01,987][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:30:02,489][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:30:02,992][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:30:03,492][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:30:03,995][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:30:04,500][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:30:05,000][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:30:05,508][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10737 tokens.
[2025-11-13 00:30:06,214][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.52%, ΔTime: 00:00:32
[2025-11-13 00:30:06,994][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:30:06,996][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:30:06,998][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:30:08,045][__main__][INFO] - Iteration 159 took 53s (32.34% Gen, 65.70% Train). Generation: 17s, Training: 35s. Estimated remaining time: 42h 14m 23s. Estimated total time: 44h 34m 49s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 9s, 500 more iterations: 7h 25m 48s.
[2025-11-13 00:30:08,047][__main__][INFO] - Starting iteration 159.
[2025-11-13 00:30:08,542][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:30:08,542][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:30:12,307][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:30:25,949][__main__][INFO] - Number of regex retries in iteration 159: 1
[2025-11-13 00:30:25,950][__main__][INFO] - agents played in iteration 159 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:30:26,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:30:26,824][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:30:26,849][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:30:26,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:30:26,873][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:30:26,874][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:30:27,502][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:30:27,969][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:30:28,478][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:30:28,980][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:30:29,481][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:30:29,982][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:30:30,489][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:30:30,989][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:30:31,489][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:30:31,991][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:30:32,486][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:30:32,985][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:30:33,482][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:30:33,982][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:30:34,489][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:30:34,986][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:30:35,480][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:30:35,974][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:30:36,467][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:30:36,974][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:30:37,468][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:30:37,966][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:30:38,482][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:30:38,980][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:30:39,477][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:30:39,978][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:30:40,476][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:30:40,989][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:30:41,490][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:30:41,991][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:30:42,492][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:30:42,991][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:30:43,492][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:30:43,991][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:30:44,491][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:30:44,996][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:30:45,494][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:30:45,993][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:30:46,490][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:30:46,989][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:30:47,495][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:30:47,995][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:30:48,494][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:30:48,997][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:30:49,500][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:30:50,003][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:30:50,506][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:30:51,006][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:30:51,508][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:30:52,007][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:30:52,507][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:30:53,008][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:30:53,505][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:30:54,004][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:30:54,503][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:30:55,000][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:30:55,502][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:30:56,003][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:30:56,505][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:30:57,009][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:30:57,511][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:30:58,009][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:30:58,511][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:30:59,009][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:30:59,512][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10695 tokens.
[2025-11-13 00:31:00,222][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.36%, ΔTime: 00:00:32
[2025-11-13 00:31:00,974][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:31:00,976][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:31:00,978][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:31:02,158][__main__][INFO] - Iteration 160 took 53s (32.47% Gen, 65.33% Train). Generation: 17s, Training: 35s. Estimated remaining time: 42h 19m 29s. Estimated total time: 44h 40m 49s. Time estimates for 10 more iterations: 8m 56s, 100 more iterations: 1h 29m 21s, 500 more iterations: 7h 26m 48s.
[2025-11-13 00:31:02,159][__main__][INFO] - Starting iteration 160.
[2025-11-13 00:31:02,633][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 00:31:02,634][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:31:06,011][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:31:19,165][__main__][INFO] - Number of regex retries in iteration 160: 1
[2025-11-13 00:31:19,166][__main__][INFO] - agents played in iteration 160 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:31:19,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:31:19,971][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:31:19,996][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:31:20,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:31:20,019][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:31:20,020][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:31:20,629][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:31:21,084][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:31:21,585][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:31:22,083][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:31:22,581][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:31:23,096][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:31:23,590][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:31:24,090][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:31:24,599][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:31:25,098][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:31:25,595][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:31:26,095][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:31:26,590][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:31:27,098][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:31:27,596][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:31:28,092][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:31:28,590][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:31:29,087][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:31:29,592][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:31:30,088][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:31:30,588][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:31:31,089][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:31:31,588][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:31:32,089][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:31:32,589][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:31:33,087][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:31:33,596][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:31:34,097][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:31:34,598][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:31:35,099][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:31:35,597][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:31:36,104][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:31:36,600][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:31:37,100][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:31:37,599][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:31:38,097][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:31:38,596][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:31:39,094][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:31:39,589][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:31:40,102][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:31:40,599][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:31:41,100][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:31:41,604][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:31:42,109][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:31:42,611][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:31:43,109][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:31:43,605][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:31:44,106][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:31:44,606][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:31:45,107][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:31:45,611][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:31:46,110][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:31:46,613][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:31:47,111][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:31:47,612][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:31:48,110][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:31:48,616][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:31:49,117][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:31:49,621][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:31:50,122][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:31:50,626][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:31:51,127][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:31:51,630][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:31:52,132][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:31:52,634][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10727 tokens.
[2025-11-13 00:31:53,348][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 00:31:54,078][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:31:54,080][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:31:54,082][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:31:56,062][__main__][INFO] - Iteration 161 took 53s (30.94% Gen, 65.35% Train). Generation: 16s, Training: 34s. Estimated remaining time: 42h 9m 13s. Estimated total time: 44h 31m 28s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 2s, 500 more iterations: 7h 25m 14s.
[2025-11-13 00:31:56,064][__main__][INFO] - Starting iteration 161.
[2025-11-13 00:31:56,553][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:31:56,553][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:32:14,382][__main__][INFO] - Number of regex retries in iteration 161: 0
[2025-11-13 00:32:14,382][__main__][INFO] - agents played in iteration 161 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:32:15,257][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:32:15,282][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:32:15,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:32:15,329][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:32:15,330][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:32:15,330][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:32:15,963][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:32:16,415][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:32:16,923][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:32:17,422][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:32:17,922][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:32:18,422][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:32:18,921][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:32:19,425][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:32:19,924][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:32:20,425][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:32:20,923][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:32:21,424][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:32:21,925][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:32:22,428][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:32:22,927][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:32:23,432][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:32:23,937][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:32:24,440][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:32:24,938][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:32:25,440][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:32:25,944][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:32:26,446][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:32:26,947][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:32:27,448][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:32:27,949][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:32:28,451][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:32:28,945][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:32:29,444][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:32:29,942][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:32:30,442][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:32:30,940][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:32:31,436][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:32:31,938][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:32:32,439][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:32:32,937][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:32:33,436][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:32:33,937][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:32:34,440][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:32:34,940][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:32:35,438][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:32:35,940][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:32:36,442][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:32:36,940][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:32:37,442][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:32:37,945][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:32:38,444][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:32:38,944][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:32:39,444][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:32:39,947][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:32:40,449][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:32:40,952][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:32:41,451][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:32:41,951][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:32:42,451][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:32:42,951][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:32:43,450][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:32:43,949][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:32:44,450][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:32:44,950][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:32:45,451][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:32:45,949][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:32:46,446][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:32:46,946][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:32:47,441][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:32:47,938][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10710 tokens.
[2025-11-13 00:32:48,604][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 00:32:49,378][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:32:49,379][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:32:49,381][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:32:50,453][__main__][INFO] - Iteration 162 took 53s (33.08% Gen, 64.93% Train). Generation: 17s, Training: 34s. Estimated remaining time: 42h 31m 53s. Estimated total time: 44h 55m 2s. Time estimates for 10 more iterations: 8m 59s, 100 more iterations: 1h 29m 50s, 500 more iterations: 7h 29m 10s.
[2025-11-13 00:32:50,455][__main__][INFO] - Starting iteration 162.
[2025-11-13 00:32:50,928][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:32:50,929][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:33:08,113][__main__][INFO] - Number of regex retries in iteration 162: 0
[2025-11-13 00:33:08,113][__main__][INFO] - agents played in iteration 162 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:33:08,968][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:33:08,995][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:33:09,021][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:33:09,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:33:09,044][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:33:09,045][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:33:09,660][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:33:10,114][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:33:10,614][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:33:11,112][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:33:11,609][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:33:12,108][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:33:12,606][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:33:13,113][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:33:13,616][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:33:14,122][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:33:14,628][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:33:15,134][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:33:15,636][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:33:16,138][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:33:16,638][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:33:17,143][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:33:17,644][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:33:18,144][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:33:18,645][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:33:19,146][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:33:19,649][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:33:20,148][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:33:20,648][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:33:21,145][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:33:21,643][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:33:22,141][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:33:22,635][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:33:23,133][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:33:23,634][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:33:24,133][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:33:24,630][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:33:25,134][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:33:25,632][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:33:26,130][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:33:26,630][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:33:27,129][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:33:27,634][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:33:28,135][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:33:28,637][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:33:29,137][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:33:29,635][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:33:30,139][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:33:30,642][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:33:31,147][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:33:31,649][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:33:32,153][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:33:32,657][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:33:33,160][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:33:33,667][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:33:34,170][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:33:34,670][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:33:35,168][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:33:35,668][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:33:36,169][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:33:36,672][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:33:37,171][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:33:37,672][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:33:38,174][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:33:38,674][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:33:39,173][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:33:39,672][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:33:40,171][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:33:40,671][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:33:41,170][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:33:41,667][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10769 tokens.
[2025-11-13 00:33:42,354][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.40%, ΔTime: 00:00:32
[2025-11-13 00:33:43,132][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:33:43,133][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:33:43,135][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:33:44,085][__main__][INFO] - Iteration 163 took 53s (32.33% Gen, 65.88% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 53m 50s. Estimated total time: 44h 17m 52s. Time estimates for 10 more iterations: 8m 51s, 100 more iterations: 1h 28m 35s, 500 more iterations: 7h 22m 58s.
[2025-11-13 00:33:44,089][__main__][INFO] - Starting iteration 163.
[2025-11-13 00:33:44,560][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:33:44,561][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:34:01,435][__main__][INFO] - Number of regex retries in iteration 163: 0
[2025-11-13 00:34:01,436][__main__][INFO] - agents played in iteration 163 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:34:02,255][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:34:02,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:34:02,310][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:34:02,333][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:34:02,333][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:34:02,334][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:34:02,970][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:34:03,425][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:34:03,929][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:34:04,428][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:34:04,925][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:34:05,428][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:34:05,930][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:34:06,429][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:34:06,933][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:34:07,435][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:34:07,939][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:34:08,446][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:34:08,945][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:34:09,453][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:34:09,954][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:34:10,455][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:34:10,973][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:34:11,473][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:34:11,976][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:34:12,476][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:34:12,976][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:34:13,478][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:34:13,978][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:34:14,478][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:34:14,978][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:34:15,477][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:34:15,974][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:34:16,474][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:34:16,973][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:34:17,477][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:34:17,974][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:34:18,478][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:34:18,982][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:34:19,484][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:34:19,987][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:34:20,490][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:34:20,994][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:34:21,495][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:34:21,995][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:34:22,511][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:34:23,010][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:34:23,512][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:34:24,019][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:34:24,519][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:34:25,026][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:34:25,526][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:34:26,026][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:34:26,534][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:34:27,032][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:34:27,534][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:34:28,034][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:34:28,536][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:34:29,039][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:34:29,536][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:34:30,034][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:34:30,537][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:34:31,035][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:34:31,538][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:34:32,034][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:34:32,532][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:34:33,038][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:34:33,541][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:34:34,044][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:34:34,545][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:34:35,047][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10675 tokens.
[2025-11-13 00:34:35,790][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 00:34:36,555][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:34:36,556][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:34:36,559][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:34:37,537][__main__][INFO] - Iteration 164 took 52s (31.85% Gen, 66.30% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 43m 56s. Estimated total time: 44h 8m 52s. Time estimates for 10 more iterations: 8m 49s, 100 more iterations: 1h 28m 17s, 500 more iterations: 7h 21m 28s.
[2025-11-13 00:34:37,539][__main__][INFO] - Starting iteration 164.
[2025-11-13 00:34:38,027][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:34:38,028][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:34:43,284][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:34:56,021][__main__][INFO] - Number of regex retries in iteration 164: 1
[2025-11-13 00:34:56,021][__main__][INFO] - agents played in iteration 164 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:34:56,902][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:34:56,929][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:34:56,955][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:34:56,978][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:34:56,978][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:34:56,979][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:34:57,617][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:34:58,079][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:34:58,588][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:34:59,089][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:34:59,600][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:35:00,096][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:35:00,598][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:35:01,098][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:35:01,599][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:35:02,110][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:35:02,615][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:35:03,116][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:35:03,617][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:35:04,116][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:35:04,627][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:35:05,127][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:35:05,625][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:35:06,141][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:35:06,634][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:35:07,134][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:35:07,628][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:35:08,126][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:35:08,631][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:35:09,129][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:35:09,629][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:35:10,130][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:35:10,631][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:35:11,134][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:35:11,634][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:35:12,134][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:35:12,648][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:35:13,152][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:35:13,653][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:35:14,157][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:35:14,658][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:35:15,162][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:35:15,665][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:35:16,170][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:35:16,674][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:35:17,175][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:35:17,676][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:35:18,205][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:35:18,708][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:35:19,231][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:35:19,733][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:35:20,237][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:35:20,737][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:35:21,238][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:35:21,740][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:35:22,239][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:35:22,734][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:35:23,232][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:35:23,729][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:35:24,227][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:35:24,725][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:35:25,224][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:35:25,724][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:35:26,224][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:35:26,723][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:35:27,233][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:35:27,730][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:35:28,226][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:35:28,724][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:35:29,226][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:35:29,732][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10634 tokens.
[2025-11-13 00:35:30,397][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.48%, ΔTime: 00:00:32
[2025-11-13 00:35:31,183][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:35:31,184][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:35:31,186][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:35:32,212][__main__][INFO] - Iteration 165 took 54s (33.21% Gen, 64.90% Train). Generation: 17s, Training: 35s. Estimated remaining time: 42h 43m 26s. Estimated total time: 45h 9m 17s. Time estimates for 10 more iterations: 9m 1s, 100 more iterations: 1h 30m 18s, 500 more iterations: 7h 31m 32s.
[2025-11-13 00:35:32,214][__main__][INFO] - Starting iteration 165.
[2025-11-13 00:35:32,715][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:35:32,716][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:35:40,337][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:35:48,971][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 30 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:35:49,867][__main__][INFO] - Number of regex retries in iteration 165: 2
[2025-11-13 00:35:49,868][__main__][INFO] - agents played in iteration 165 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:35:50,744][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:35:50,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:35:50,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:35:50,823][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:35:50,824][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:35:50,825][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:35:51,451][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:35:51,908][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:35:52,419][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:35:52,922][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:35:53,423][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:35:53,920][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:35:54,421][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:35:54,942][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:35:55,444][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:35:55,944][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:35:56,443][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:35:56,943][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:35:57,450][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:35:57,949][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:35:58,449][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:35:58,954][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:35:59,456][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:35:59,960][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:36:00,461][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:36:00,963][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:36:01,469][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:36:01,970][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:36:02,471][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:36:02,972][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:36:03,472][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:36:03,971][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:36:04,472][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:36:04,972][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:36:05,471][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:36:05,969][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:36:06,467][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:36:06,966][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:36:07,466][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:36:07,972][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:36:08,475][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:36:08,977][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:36:09,481][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:36:09,985][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:36:10,489][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:36:10,992][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:36:11,495][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:36:11,999][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:36:12,502][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:36:13,004][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:36:13,510][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:36:14,013][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:36:14,522][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:36:15,027][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:36:15,531][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:36:16,037][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:36:16,536][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:36:17,039][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:36:17,541][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:36:18,043][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:36:18,557][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:36:19,063][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:36:19,565][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:36:20,066][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:36:20,562][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:36:21,064][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:36:21,561][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:36:22,060][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:36:22,560][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:36:23,057][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:36:23,556][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10686 tokens.
[2025-11-13 00:36:24,216][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.39%, ΔTime: 00:00:32
[2025-11-13 00:36:25,003][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:36:25,005][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:36:25,007][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:36:26,004][__main__][INFO] - Iteration 166 took 53s (32.19% Gen, 65.94% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 57m 43s. Estimated total time: 44h 24m 27s. Time estimates for 10 more iterations: 8m 52s, 100 more iterations: 1h 28m 48s, 500 more iterations: 7h 24m 4s.
[2025-11-13 00:36:26,006][__main__][INFO] - Starting iteration 166.
[2025-11-13 00:36:26,493][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:36:26,494][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:36:40,367][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:36:42,831][__main__][INFO] - Number of regex retries in iteration 166: 1
[2025-11-13 00:36:42,832][__main__][INFO] - agents played in iteration 166 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:36:43,639][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:36:43,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:36:43,686][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:36:43,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:36:43,709][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:36:43,709][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:36:44,379][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:36:44,840][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:36:45,346][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:36:45,855][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:36:46,352][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:36:46,856][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:36:47,359][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:36:47,859][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:36:48,358][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:36:48,855][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:36:49,354][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:36:49,851][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:36:50,349][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:36:50,850][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:36:51,365][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:36:51,867][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:36:52,367][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:36:52,869][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:36:53,368][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:36:53,875][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:36:54,377][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:36:54,878][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:36:55,384][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:36:55,885][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:36:56,387][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:36:56,890][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:36:57,390][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:36:57,895][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:36:58,393][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:36:58,891][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:36:59,395][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:36:59,893][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:37:00,396][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:37:00,902][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:37:01,401][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:37:01,905][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:37:02,409][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:37:02,911][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:37:03,413][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:37:03,914][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:37:04,414][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:37:04,914][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:37:05,412][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:37:05,911][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:37:06,408][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:37:06,910][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:37:07,416][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:37:07,915][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:37:08,419][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:37:08,930][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:37:09,432][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:37:09,942][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:37:10,439][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:37:10,938][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:37:11,458][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:37:11,958][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:37:12,455][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:37:12,955][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:37:13,470][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:37:13,975][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:37:14,475][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:37:14,976][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:37:15,477][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:37:15,979][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:37:16,484][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10717 tokens.
[2025-11-13 00:37:17,154][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 00:37:17,932][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:37:17,934][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:37:17,935][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:37:18,908][__main__][INFO] - Iteration 167 took 52s (31.17% Gen, 66.97% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 13m 8s. Estimated total time: 43h 40m 45s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 21s, 500 more iterations: 7h 16m 47s.
[2025-11-13 00:37:18,910][__main__][INFO] - Starting iteration 167.
[2025-11-13 00:37:19,393][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:37:19,395][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:37:25,002][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:37:37,997][__main__][INFO] - Number of regex retries in iteration 167: 1
[2025-11-13 00:37:37,998][__main__][INFO] - agents played in iteration 167 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:37:38,861][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:37:38,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:37:38,905][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:37:38,928][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:37:38,928][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:37:38,929][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:37:39,594][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:37:40,054][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:37:40,560][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:37:41,060][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:37:41,571][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:37:42,069][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:37:42,572][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:37:43,071][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:37:43,572][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:37:44,070][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:37:44,568][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:37:45,067][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:37:45,571][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:37:46,071][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:37:46,571][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:37:47,070][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:37:47,568][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:37:48,070][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:37:48,571][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:37:49,070][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:37:49,574][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:37:50,073][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:37:50,572][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:37:51,070][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:37:51,568][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:37:52,075][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:37:52,576][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:37:53,078][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:37:53,608][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:37:54,114][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:37:54,621][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:37:55,134][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:37:55,636][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:37:56,158][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:37:56,655][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:37:57,156][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:37:57,654][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:37:58,155][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:37:58,653][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:37:59,155][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:37:59,652][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:38:00,167][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:38:00,667][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:38:01,168][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:38:01,666][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:38:02,166][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:38:02,668][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:38:03,169][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:38:03,668][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:38:04,170][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:38:04,667][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:38:05,168][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:38:05,669][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:38:06,167][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:38:06,668][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:38:07,165][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:38:07,663][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:38:08,170][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:38:08,671][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:38:09,171][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:38:09,670][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:38:10,174][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:38:10,679][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:38:11,178][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:38:11,678][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10672 tokens.
[2025-11-13 00:38:12,392][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.30%, Current % of VRAM taken: 60.56%, Block Peak % of device VRAM: 62.50%, ΔTime: 00:00:32
[2025-11-13 00:38:13,174][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:38:13,176][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:38:13,178][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:38:14,164][__main__][INFO] - Iteration 168 took 54s (33.96% Gen, 64.23% Train). Generation: 18s, Training: 35s. Estimated remaining time: 43h 10m 1s. Estimated total time: 45h 38m 34s. Time estimates for 10 more iterations: 9m 7s, 100 more iterations: 1h 31m 17s, 500 more iterations: 7h 36m 25s.
[2025-11-13 00:38:14,166][__main__][INFO] - Starting iteration 168.
[2025-11-13 00:38:14,655][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:38:14,656][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:38:31,762][__main__][INFO] - Number of regex retries in iteration 168: 0
[2025-11-13 00:38:31,763][__main__][INFO] - agents played in iteration 168 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:38:32,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:38:32,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:38:32,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:38:32,693][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:38:32,694][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:38:32,694][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:38:33,344][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:38:33,801][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:38:34,311][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:38:34,811][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:38:35,314][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:38:35,820][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:38:36,323][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:38:36,827][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:38:37,330][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:38:37,830][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:38:38,333][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:38:38,834][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:38:39,334][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:38:39,831][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:38:40,335][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:38:40,864][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:38:41,364][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:38:41,860][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:38:42,364][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:38:42,864][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:38:43,367][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:38:43,865][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:38:44,364][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:38:44,876][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:38:45,380][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:38:45,885][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:38:46,386][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:38:46,887][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:38:47,389][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:38:47,888][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:38:48,387][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:38:48,889][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:38:49,390][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:38:49,893][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:38:50,394][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:38:50,897][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:38:51,401][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:38:51,904][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:38:52,410][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:38:52,911][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:38:53,412][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:38:53,915][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:38:54,434][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:38:54,940][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:38:55,441][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:38:55,943][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:38:56,458][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:38:56,960][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:38:57,460][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:38:57,961][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:38:58,458][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:38:58,967][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:38:59,468][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:38:59,966][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:39:00,479][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:39:00,975][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:39:01,478][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:39:01,975][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:39:02,477][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:39:02,984][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:39:03,484][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:39:03,982][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:39:04,486][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:39:04,990][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:39:05,492][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10716 tokens.
[2025-11-13 00:39:06,151][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 00:39:06,926][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:39:06,928][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:39:06,929][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:39:07,859][__main__][INFO] - Iteration 169 took 53s (32.15% Gen, 66.10% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 50m 45s. Estimated total time: 44h 20m 11s. Time estimates for 10 more iterations: 8m 52s, 100 more iterations: 1h 28m 40s, 500 more iterations: 7h 23m 21s.
[2025-11-13 00:39:07,861][__main__][INFO] - Starting iteration 169.
[2025-11-13 00:39:08,340][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:39:08,341][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:39:19,149][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:39:24,604][__main__][INFO] - Number of regex retries in iteration 169: 1
[2025-11-13 00:39:24,605][__main__][INFO] - agents played in iteration 169 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:39:25,383][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:39:25,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:39:25,428][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:39:25,450][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:39:25,450][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:39:25,451][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:39:26,126][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:39:26,586][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:39:27,089][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:39:27,593][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:39:28,089][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:39:28,593][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:39:29,097][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:39:29,597][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:39:30,095][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:39:30,599][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:39:31,100][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:39:31,606][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:39:32,104][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:39:32,606][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:39:33,109][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:39:33,610][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:39:34,115][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:39:34,612][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:39:35,113][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:39:35,616][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:39:36,115][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:39:36,613][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:39:37,115][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:39:37,616][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:39:38,117][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:39:38,616][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:39:39,116][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:39:39,616][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:39:40,113][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:39:40,616][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:39:41,114][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:39:41,616][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:39:42,120][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:39:42,619][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:39:43,117][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:39:43,616][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:39:44,115][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:39:44,619][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:39:45,119][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:39:45,616][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:39:46,112][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:39:46,610][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:39:47,112][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:39:47,610][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:39:48,113][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:39:48,614][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:39:49,111][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:39:49,609][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:39:50,109][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:39:50,611][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:39:51,111][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:39:51,608][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:39:52,107][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:39:52,608][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:39:53,107][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:39:53,607][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:39:54,106][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:39:54,604][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:39:55,106][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:39:55,605][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:39:56,102][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:39:56,607][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:39:57,109][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:39:57,607][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:39:58,107][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10676 tokens.
[2025-11-13 00:39:58,822][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 00:39:59,572][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:39:59,574][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:39:59,575][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:40:00,500][__main__][INFO] - Iteration 170 took 52s (31.18% Gen, 67.04% Train). Generation: 16s, Training: 34s. Estimated remaining time: 40h 57m 41s. Estimated total time: 43h 28m 0s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 56s, 500 more iterations: 7h 14m 40s.
[2025-11-13 00:40:00,502][__main__][INFO] - Starting iteration 170.
[2025-11-13 00:40:00,986][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 00:40:00,986][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:40:17,208][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:40:18,019][__main__][INFO] - Number of regex retries in iteration 170: 1
[2025-11-13 00:40:18,020][__main__][INFO] - agents played in iteration 170 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:40:18,875][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:40:18,900][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:40:18,928][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:40:18,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:40:18,952][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:40:18,953][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:40:19,601][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:40:20,075][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:40:20,585][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:40:21,086][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:40:21,590][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:40:22,092][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:40:22,598][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:40:23,101][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:40:23,601][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:40:24,100][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:40:24,601][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:40:25,101][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:40:25,601][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:40:26,103][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:40:26,602][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:40:27,104][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:40:27,606][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:40:28,110][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:40:28,610][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:40:29,110][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:40:29,608][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:40:30,107][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:40:30,607][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:40:31,103][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:40:31,604][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:40:32,105][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:40:32,608][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:40:33,111][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:40:33,613][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:40:34,117][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:40:34,619][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:40:35,121][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:40:35,629][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:40:36,132][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:40:36,632][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:40:37,135][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:40:37,637][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:40:38,139][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:40:38,640][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:40:39,140][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:40:39,640][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:40:40,141][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:40:40,640][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:40:41,145][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:40:41,645][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:40:42,144][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:40:42,642][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:40:43,140][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:40:43,641][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:40:44,138][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:40:44,633][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:40:45,130][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:40:45,628][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:40:46,126][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:40:46,620][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:40:47,118][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:40:47,617][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:40:48,117][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:40:48,616][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:40:49,121][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:40:49,620][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:40:50,122][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:40:50,622][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:40:51,122][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:40:51,623][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10704 tokens.
[2025-11-13 00:40:52,307][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 00:40:53,102][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:40:53,103][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:40:53,105][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:40:54,871][__main__][INFO] - Iteration 171 took 53s (31.61% Gen, 65.11% Train). Generation: 17s, Training: 35s. Estimated remaining time: 42h 23m 5s. Estimated total time: 44h 54m 18s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 48s, 500 more iterations: 7h 29m 3s.
[2025-11-13 00:40:54,874][__main__][INFO] - Starting iteration 171.
[2025-11-13 00:40:55,365][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:40:55,366][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:41:00,047][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 30 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:41:12,148][__main__][INFO] - Number of regex retries in iteration 171: 1
[2025-11-13 00:41:12,148][__main__][INFO] - agents played in iteration 171 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:41:12,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:41:13,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:41:13,025][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:41:13,046][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:41:13,047][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:41:13,048][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:41:13,743][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:41:14,202][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:41:14,709][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:41:15,217][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:41:15,716][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:41:16,219][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:41:16,726][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:41:17,228][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:41:17,732][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:41:18,229][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:41:18,728][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:41:19,228][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:41:19,730][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:41:20,233][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:41:20,731][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:41:21,228][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:41:21,725][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:41:22,226][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:41:22,726][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:41:23,226][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:41:23,732][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:41:24,231][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:41:24,737][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:41:25,237][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:41:25,745][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:41:26,249][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:41:26,755][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:41:27,260][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:41:27,767][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:41:28,274][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:41:28,778][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:41:29,279][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:41:29,794][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:41:30,295][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:41:30,797][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:41:31,299][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:41:31,798][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:41:32,305][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:41:32,805][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:41:33,305][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:41:33,814][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:41:34,313][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:41:34,818][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:41:35,314][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:41:35,812][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:41:36,314][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:41:36,813][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:41:37,308][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:41:37,808][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:41:38,305][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:41:38,808][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:41:39,307][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:41:39,806][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:41:40,316][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:41:40,814][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:41:41,310][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:41:41,810][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:41:42,310][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:41:42,815][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:41:43,312][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:41:43,809][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:41:44,312][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:41:44,817][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:41:45,313][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:41:45,811][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10652 tokens.
[2025-11-13 00:41:46,469][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 00:41:47,258][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:41:47,260][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:41:47,262][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:41:48,213][__main__][INFO] - Iteration 172 took 52s (31.76% Gen, 66.44% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 30m 20s. Estimated total time: 44h 2m 27s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 4s, 500 more iterations: 7h 20m 24s.
[2025-11-13 00:41:48,215][__main__][INFO] - Starting iteration 172.
[2025-11-13 00:41:48,680][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:41:48,680][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:41:52,423][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:41:53,710][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:42:04,955][__main__][INFO] - Number of regex retries in iteration 172: 2
[2025-11-13 00:42:04,956][__main__][INFO] - agents played in iteration 172 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:42:05,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:42:05,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:42:05,801][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:42:05,826][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:42:05,827][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:42:05,828][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:42:06,491][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:42:06,949][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:42:07,456][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:42:07,957][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:42:08,460][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:42:08,966][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:42:09,468][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:42:09,970][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:42:10,484][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:42:10,982][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:42:11,486][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:42:11,985][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:42:12,486][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:42:12,986][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:42:13,486][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:42:13,986][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:42:14,486][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:42:14,985][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:42:15,497][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:42:15,992][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:42:16,490][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:42:17,005][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:42:17,508][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:42:18,010][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:42:18,512][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:42:19,014][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:42:19,525][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:42:20,025][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:42:20,525][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:42:21,027][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:42:21,527][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:42:22,031][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:42:22,531][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:42:23,031][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:42:23,535][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:42:24,034][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:42:24,535][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:42:25,032][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:42:25,530][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:42:26,030][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:42:26,528][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:42:27,029][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:42:27,528][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:42:28,027][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:42:28,529][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:42:29,027][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:42:29,527][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:42:30,033][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:42:30,534][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:42:31,029][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:42:31,526][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:42:32,021][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:42:32,522][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:42:33,019][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:42:33,516][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:42:34,014][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:42:34,513][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:42:35,011][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:42:35,510][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:42:36,006][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:42:36,510][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:42:37,006][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:42:37,505][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:42:38,021][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:42:38,517][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10756 tokens.
[2025-11-13 00:42:39,174][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 00:42:39,941][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:42:39,942][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:42:39,944][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:42:40,927][__main__][INFO] - Iteration 173 took 52s (31.15% Gen, 66.97% Train). Generation: 16s, Training: 34s. Estimated remaining time: 40h 59m 24s. Estimated total time: 43h 32m 24s. Time estimates for 10 more iterations: 8m 42s, 100 more iterations: 1h 27m 4s, 500 more iterations: 7h 15m 24s.
[2025-11-13 00:42:40,930][__main__][INFO] - Starting iteration 173.
[2025-11-13 00:42:41,415][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:42:41,416][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:42:57,146][__main__][INFO] - Number of regex retries in iteration 173: 0
[2025-11-13 00:42:57,147][__main__][INFO] - agents played in iteration 173 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:42:57,956][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:42:57,981][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:42:58,005][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:42:58,027][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:42:58,028][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:42:58,028][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:42:58,734][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:42:59,201][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:42:59,710][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:43:00,227][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:43:00,729][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:43:01,228][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:43:01,729][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:43:02,229][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:43:02,734][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:43:03,235][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:43:03,733][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:43:04,234][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:43:04,731][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:43:05,230][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:43:05,728][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:43:06,231][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:43:06,733][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:43:07,231][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:43:07,727][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:43:08,226][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:43:08,723][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:43:09,223][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:43:09,720][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:43:10,216][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:43:10,727][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:43:11,228][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:43:11,729][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:43:12,228][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:43:12,729][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:43:13,236][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:43:13,735][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:43:14,236][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:43:14,735][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:43:15,235][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:43:15,736][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:43:16,235][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:43:16,736][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:43:17,238][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:43:17,734][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:43:18,232][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:43:18,731][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:43:19,227][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:43:19,724][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:43:20,223][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:43:20,720][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:43:21,231][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:43:21,728][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:43:22,226][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:43:22,723][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:43:23,222][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:43:23,727][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:43:24,227][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:43:24,725][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:43:25,225][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:43:25,724][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:43:26,223][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:43:26,723][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:43:27,218][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:43:27,719][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:43:28,219][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:43:28,716][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:43:29,219][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:43:29,716][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:43:30,217][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:43:30,720][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10739 tokens.
[2025-11-13 00:43:31,373][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 00:43:32,166][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:43:32,168][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:43:32,169][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:43:33,111][__main__][INFO] - Iteration 174 took 51s (30.43% Gen, 67.75% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 30m 55s. Estimated total time: 43h 4m 47s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 9s, 500 more iterations: 7h 10m 47s.
[2025-11-13 00:43:33,113][__main__][INFO] - Starting iteration 174.
[2025-11-13 00:43:33,593][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:43:33,594][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:43:39,224][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:43:46,213][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:43:49,761][__main__][INFO] - Number of regex retries in iteration 174: 2
[2025-11-13 00:43:49,762][__main__][INFO] - agents played in iteration 174 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:43:50,630][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:43:50,657][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:43:50,683][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:43:50,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:43:50,706][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:43:50,707][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:43:51,386][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:43:51,844][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:43:52,349][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:43:52,853][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:43:53,350][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:43:53,846][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:43:54,342][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:43:54,841][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:43:55,341][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:43:55,839][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:43:56,339][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:43:56,846][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:43:57,346][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:43:57,848][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:43:58,345][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:43:58,845][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:43:59,348][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:43:59,847][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:44:00,344][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:44:00,846][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:44:01,344][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:44:01,842][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:44:02,339][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:44:02,836][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:44:03,339][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:44:03,841][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:44:04,337][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:44:04,845][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:44:05,344][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:44:05,844][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:44:06,342][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:44:06,838][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:44:07,337][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:44:07,837][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:44:08,337][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:44:08,846][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:44:09,346][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:44:09,847][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:44:10,344][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:44:10,841][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:44:11,345][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:44:11,843][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:44:12,337][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:44:12,843][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:44:13,342][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:44:13,843][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:44:14,347][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:44:14,845][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:44:15,342][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:44:15,841][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:44:16,339][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:44:16,839][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:44:17,337][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:44:17,840][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:44:18,338][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:44:18,835][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:44:19,334][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:44:19,835][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:44:20,331][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:44:20,831][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:44:21,333][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:44:21,840][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:44:22,341][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:44:22,856][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:44:23,354][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10714 tokens.
[2025-11-13 00:44:24,034][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 00:44:24,810][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:44:24,811][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:44:24,813][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:44:25,737][__main__][INFO] - Iteration 175 took 52s (31.01% Gen, 67.22% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 52m 30s. Estimated total time: 43h 27m 14s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 54s, 500 more iterations: 7h 14m 32s.
[2025-11-13 00:44:25,739][__main__][INFO] - Starting iteration 175.
[2025-11-13 00:44:26,229][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:44:26,230][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:44:39,831][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:44:42,482][__main__][INFO] - Number of regex retries in iteration 175: 1
[2025-11-13 00:44:42,482][__main__][INFO] - agents played in iteration 175 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:44:43,263][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:44:43,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:44:43,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:44:43,338][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:44:43,339][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:44:43,340][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:44:43,996][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:44:44,453][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:44:44,957][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:44:45,462][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:44:45,964][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:44:46,466][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:44:46,969][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:44:47,469][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:44:47,970][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:44:48,473][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:44:48,976][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:44:49,480][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:44:49,982][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:44:50,481][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:44:50,983][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:44:51,482][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:44:51,985][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:44:52,485][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:44:52,988][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:44:53,487][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:44:53,985][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:44:54,486][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:44:54,985][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:44:55,486][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:44:55,989][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:44:56,490][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:44:56,992][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:44:57,494][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:44:57,993][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:44:58,501][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:44:59,004][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:44:59,505][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:45:00,018][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:45:00,519][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:45:01,025][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:45:01,526][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:45:02,025][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:45:02,546][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:45:03,042][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:45:03,542][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:45:04,041][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:45:04,539][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:45:05,044][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:45:05,543][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:45:06,043][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:45:06,544][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:45:07,042][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:45:07,541][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:45:08,038][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:45:08,536][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:45:09,049][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:45:09,547][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:45:10,046][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:45:10,545][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:45:11,045][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:45:11,549][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:45:12,046][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:45:12,547][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:45:13,053][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:45:13,551][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:45:14,053][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:45:14,553][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:45:15,053][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:45:15,556][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:45:16,057][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10761 tokens.
[2025-11-13 00:45:16,746][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 00:45:17,499][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:45:17,501][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:45:17,504][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:45:18,396][__main__][INFO] - Iteration 176 took 52s (31.15% Gen, 67.14% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 52m 42s. Estimated total time: 43h 28m 19s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 56s, 500 more iterations: 7h 14m 43s.
[2025-11-13 00:45:18,398][__main__][INFO] - Starting iteration 176.
[2025-11-13 00:45:18,878][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:45:18,879][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:45:35,202][__main__][INFO] - Number of regex retries in iteration 176: 0
[2025-11-13 00:45:35,202][__main__][INFO] - agents played in iteration 176 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:45:35,983][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:45:36,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:45:36,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:45:36,058][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:45:36,058][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:45:36,059][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:45:36,794][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:45:37,250][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:45:37,755][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:45:38,254][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:45:38,755][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:45:39,264][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:45:39,767][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:45:40,271][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:45:40,770][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:45:41,268][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:45:41,768][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:45:42,266][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:45:42,762][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:45:43,265][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:45:43,764][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:45:44,263][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:45:44,759][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:45:45,256][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:45:45,753][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:45:46,250][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:45:46,747][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:45:47,250][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:45:47,747][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:45:48,249][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:45:48,752][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:45:49,251][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:45:49,753][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:45:50,255][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:45:50,752][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:45:51,259][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:45:51,760][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:45:52,260][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:45:52,758][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:45:53,257][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:45:53,762][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:45:54,257][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:45:54,754][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:45:55,263][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:45:55,764][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:45:56,262][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:45:56,759][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:45:57,257][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:45:57,761][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:45:58,263][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:45:58,764][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:45:59,266][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:45:59,766][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:46:00,271][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:46:00,766][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:46:01,269][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:46:01,775][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:46:02,274][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:46:02,775][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:46:03,275][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:46:03,776][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:46:04,280][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:46:04,790][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:46:05,292][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:46:05,794][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:46:06,297][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:46:06,800][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:46:07,306][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:46:07,807][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:46:08,308][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:46:08,808][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10740 tokens.
[2025-11-13 00:46:09,493][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 00:46:10,249][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:46:10,250][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:46:10,252][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:46:11,206][__main__][INFO] - Iteration 177 took 52s (31.19% Gen, 66.98% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 59m 54s. Estimated total time: 43h 36m 24s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 12s, 500 more iterations: 7h 16m 4s.
[2025-11-13 00:46:11,208][__main__][INFO] - Starting iteration 177.
[2025-11-13 00:46:11,687][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:46:11,687][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:46:20,982][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:46:29,117][__main__][INFO] - Number of regex retries in iteration 177: 1
[2025-11-13 00:46:29,117][__main__][INFO] - agents played in iteration 177 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:46:29,956][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:46:29,983][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:46:30,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:46:30,032][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:46:30,032][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:46:30,033][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:46:30,673][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:46:31,128][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:46:31,631][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:46:32,149][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:46:32,652][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:46:33,153][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:46:33,653][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:46:34,151][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:46:34,650][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:46:35,152][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:46:35,652][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:46:36,166][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:46:36,665][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:46:37,165][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:46:37,666][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:46:38,165][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:46:38,667][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:46:39,188][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:46:39,690][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:46:40,192][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:46:40,696][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:46:41,201][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:46:41,703][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:46:42,206][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:46:42,707][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:46:43,206][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:46:43,706][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:46:44,207][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:46:44,703][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:46:45,204][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:46:45,699][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:46:46,196][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:46:46,693][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:46:47,192][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:46:47,691][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:46:48,190][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:46:48,690][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:46:49,195][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:46:49,694][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:46:50,191][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:46:50,692][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:46:51,187][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:46:51,684][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:46:52,180][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:46:52,678][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:46:53,181][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:46:53,676][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:46:54,171][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:46:54,670][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:46:55,171][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:46:55,670][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:46:56,169][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:46:56,673][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:46:57,186][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:46:57,689][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:46:58,191][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:46:58,701][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:46:59,207][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:46:59,708][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:47:00,212][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:47:00,715][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:47:01,215][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:47:01,719][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:47:02,223][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:47:02,734][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10688 tokens.
[2025-11-13 00:47:03,405][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 00:47:04,183][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:47:04,185][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:47:04,186][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:47:05,119][__main__][INFO] - Iteration 178 took 53s (32.62% Gen, 65.63% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 54m 15s. Estimated total time: 44h 31m 38s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 3s, 500 more iterations: 7h 25m 16s.
[2025-11-13 00:47:05,121][__main__][INFO] - Starting iteration 178.
[2025-11-13 00:47:05,603][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:47:05,603][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:47:22,174][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:47:23,040][__main__][INFO] - Number of regex retries in iteration 178: 1
[2025-11-13 00:47:23,041][__main__][INFO] - agents played in iteration 178 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:47:23,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:47:23,840][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:47:23,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:47:23,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:47:23,884][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:47:23,885][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:47:24,502][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:47:24,966][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:47:25,469][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:47:25,969][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:47:26,469][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:47:26,968][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:47:27,478][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:47:27,976][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:47:28,474][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:47:28,974][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:47:29,471][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:47:29,975][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:47:30,475][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:47:30,972][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:47:31,472][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:47:31,970][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:47:32,472][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:47:32,977][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:47:33,479][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:47:33,985][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:47:34,487][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:47:34,990][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:47:35,492][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:47:35,993][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:47:36,495][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:47:36,995][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:47:37,495][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:47:37,998][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:47:38,499][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:47:39,000][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:47:39,501][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:47:40,000][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:47:40,502][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:47:40,999][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:47:41,498][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:47:41,998][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:47:42,502][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:47:43,000][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:47:43,500][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:47:43,994][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:47:44,497][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:47:44,996][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:47:45,494][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:47:45,994][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:47:46,493][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:47:46,992][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:47:47,491][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:47:47,987][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:47:48,489][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:47:48,989][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:47:49,487][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:47:49,985][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:47:50,487][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:47:50,996][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:47:51,495][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:47:51,993][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:47:52,493][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:47:52,993][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:47:53,490][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:47:53,992][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:47:54,490][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:47:54,994][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:47:55,496][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:47:55,998][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:47:56,499][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10721 tokens.
[2025-11-13 00:47:57,196][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.39%, ΔTime: 00:00:32
[2025-11-13 00:47:57,973][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:47:57,975][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:47:57,977][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:47:58,895][__main__][INFO] - Iteration 179 took 53s (32.72% Gen, 65.55% Train). Generation: 17s, Training: 34s. Estimated remaining time: 41h 46m 20s. Estimated total time: 44h 24m 37s. Time estimates for 10 more iterations: 8m 52s, 100 more iterations: 1h 28m 49s, 500 more iterations: 7h 24m 6s.
[2025-11-13 00:47:58,897][__main__][INFO] - Starting iteration 179.
[2025-11-13 00:47:59,384][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:47:59,385][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:48:06,374][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:48:06,988][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:48:09,131][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:48:16,770][__main__][INFO] - Number of regex retries in iteration 179: 3
[2025-11-13 00:48:16,771][__main__][INFO] - agents played in iteration 179 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:48:17,551][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:48:17,579][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:48:17,605][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:48:17,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:48:17,628][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:48:17,629][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:48:18,260][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:48:18,714][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:48:19,214][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:48:19,714][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:48:20,213][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:48:20,712][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:48:21,211][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:48:21,710][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:48:22,210][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:48:22,711][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:48:23,208][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:48:23,707][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:48:24,203][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:48:24,703][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:48:25,206][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:48:25,706][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:48:26,226][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:48:26,732][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:48:27,236][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:48:27,748][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:48:28,249][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:48:28,763][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:48:29,262][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:48:29,768][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:48:30,269][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:48:30,770][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:48:31,279][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:48:31,779][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:48:32,278][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:48:32,784][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:48:33,284][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:48:33,785][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:48:34,288][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:48:34,787][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:48:35,290][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:48:35,789][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:48:36,287][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:48:36,786][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:48:37,286][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:48:37,782][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:48:38,280][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:48:38,775][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:48:39,291][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:48:39,788][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:48:40,288][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:48:40,790][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:48:41,290][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:48:41,793][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:48:42,293][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:48:42,794][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:48:43,300][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:48:43,802][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:48:44,303][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:48:44,803][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:48:45,300][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:48:45,805][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:48:46,302][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:48:46,802][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:48:47,305][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:48:47,805][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:48:48,308][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:48:48,809][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:48:49,312][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:48:49,814][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:48:50,316][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10730 tokens.
[2025-11-13 00:48:51,029][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 00:48:51,775][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:48:51,777][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:48:51,781][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:48:52,665][__main__][INFO] - Iteration 180 took 53s (32.63% Gen, 65.71% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 44m 55s. Estimated total time: 44h 24m 6s. Time estimates for 10 more iterations: 8m 52s, 100 more iterations: 1h 28m 48s, 500 more iterations: 7h 24m 1s.
[2025-11-13 00:48:52,668][__main__][INFO] - Starting iteration 180.
[2025-11-13 00:48:53,145][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 00:48:53,146][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:49:08,649][__main__][INFO] - Number of regex retries in iteration 180: 0
[2025-11-13 00:49:08,650][__main__][INFO] - agents played in iteration 180 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:49:09,491][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:49:09,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:49:09,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:49:09,567][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:49:09,568][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:49:09,569][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:49:10,187][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:49:10,641][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:49:11,143][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:49:11,641][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:49:12,139][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:49:12,637][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:49:13,134][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:49:13,632][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:49:14,131][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:49:14,630][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:49:15,127][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:49:15,627][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:49:16,125][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:49:16,623][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:49:17,118][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:49:17,622][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:49:18,123][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:49:18,628][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:49:19,134][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:49:19,636][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:49:20,138][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:49:20,644][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:49:21,145][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:49:21,649][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:49:22,153][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:49:22,653][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:49:23,150][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:49:23,650][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:49:24,155][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:49:24,650][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:49:25,149][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:49:25,648][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:49:26,146][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:49:26,646][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:49:27,146][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:49:27,645][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:49:28,147][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:49:28,646][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:49:29,146][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:49:29,646][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:49:30,145][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:49:30,647][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:49:31,146][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:49:31,649][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:49:32,149][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:49:32,649][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:49:33,154][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:49:33,654][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:49:34,156][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:49:34,667][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:49:35,168][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:49:35,672][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:49:36,171][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:49:36,671][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:49:37,172][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:49:37,672][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:49:38,170][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:49:38,671][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:49:39,174][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:49:39,674][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:49:40,173][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:49:40,673][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:49:41,188][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:49:41,690][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:49:42,193][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10767 tokens.
[2025-11-13 00:49:42,892][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.28%, Current % of VRAM taken: 60.54%, Block Peak % of device VRAM: 62.52%, ΔTime: 00:00:32
[2025-11-13 00:49:43,664][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:49:43,665][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:49:43,667][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:49:45,670][__main__][INFO] - Iteration 181 took 52s (29.52% Gen, 66.67% Train). Generation: 15s, Training: 35s. Estimated remaining time: 41h 6m 10s. Estimated total time: 43h 46m 14s. Time estimates for 10 more iterations: 8m 45s, 100 more iterations: 1h 27m 32s, 500 more iterations: 7h 17m 42s.
[2025-11-13 00:49:45,672][__main__][INFO] - Starting iteration 181.
[2025-11-13 00:49:46,174][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:49:46,174][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:49:58,337][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:50:02,846][__main__][INFO] - Number of regex retries in iteration 181: 1
[2025-11-13 00:50:02,847][__main__][INFO] - agents played in iteration 181 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:50:03,626][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:50:03,648][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:50:03,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:50:03,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:50:03,693][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:50:03,693][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:50:04,316][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:50:04,777][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:50:05,280][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:50:05,781][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:50:06,286][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:50:06,787][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:50:07,287][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:50:07,788][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:50:08,289][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:50:08,790][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:50:09,287][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:50:09,785][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:50:10,291][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:50:10,792][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:50:11,293][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:50:11,793][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:50:12,296][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:50:12,799][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:50:13,301][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:50:13,803][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:50:14,302][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:50:14,801][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:50:15,317][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:50:15,816][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:50:16,313][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:50:16,814][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:50:17,316][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:50:17,825][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:50:18,324][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:50:18,824][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:50:19,323][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:50:19,821][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:50:20,319][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:50:20,817][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:50:21,314][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:50:21,828][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:50:22,328][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:50:22,830][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:50:23,332][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:50:23,832][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:50:24,334][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:50:24,831][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:50:25,331][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:50:25,839][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:50:26,339][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:50:26,846][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:50:27,347][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:50:27,848][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:50:28,353][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:50:28,853][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:50:29,356][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:50:29,859][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:50:30,361][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:50:30,862][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:50:31,363][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:50:31,863][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:50:32,363][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:50:32,870][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:50:33,373][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:50:33,878][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:50:34,381][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:50:34,882][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:50:35,387][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:50:35,885][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:50:36,386][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10695 tokens.
[2025-11-13 00:50:37,054][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.32%, Current % of VRAM taken: 60.58%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 00:50:37,834][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:50:37,836][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:50:37,838][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:50:38,764][__main__][INFO] - Iteration 182 took 52s (31.70% Gen, 66.53% Train). Generation: 16s, Training: 34s. Estimated remaining time: 41h 8m 35s. Estimated total time: 43h 49m 32s. Time estimates for 10 more iterations: 8m 45s, 100 more iterations: 1h 27m 39s, 500 more iterations: 7h 18m 15s.
[2025-11-13 00:50:38,766][__main__][INFO] - Starting iteration 182.
[2025-11-13 00:50:39,259][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:50:39,259][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:50:44,897][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:50:46,558][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:50:56,111][__main__][INFO] - Number of regex retries in iteration 182: 2
[2025-11-13 00:50:56,112][__main__][INFO] - agents played in iteration 182 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:50:56,886][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:50:56,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:50:56,935][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:50:56,957][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.43%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:50:56,957][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:50:56,958][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:50:57,590][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:50:58,046][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:50:58,548][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:50:59,052][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:50:59,548][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:51:00,049][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:51:00,547][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:51:01,042][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:51:01,539][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:51:02,040][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:51:02,542][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:51:03,042][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:51:03,544][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:51:04,046][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:51:04,549][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:51:05,049][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:51:05,551][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:51:06,053][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:51:06,552][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:51:07,052][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:51:07,554][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:51:08,052][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:51:08,551][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:51:09,055][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:51:09,555][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:51:10,053][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:51:10,554][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:51:11,052][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:51:11,548][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:51:12,047][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:51:12,547][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:51:13,048][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:51:13,548][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:51:14,052][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:51:14,557][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:51:15,063][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:51:15,568][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:51:16,069][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:51:16,575][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:51:17,083][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:51:17,588][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:51:18,090][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:51:18,590][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:51:19,092][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:51:19,595][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:51:20,097][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:51:20,600][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:51:21,114][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:51:21,616][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:51:22,130][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:51:22,632][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:51:23,131][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:51:23,648][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:51:24,148][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:51:24,652][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:51:25,153][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:51:25,655][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:51:26,159][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:51:26,659][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:51:27,158][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:51:27,655][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:51:28,155][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:51:28,658][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:51:29,161][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:51:29,661][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10742 tokens.
[2025-11-13 00:51:30,346][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.29%, Current % of VRAM taken: 60.55%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 00:51:31,104][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:51:31,105][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:51:31,107][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:51:32,110][__main__][INFO] - Iteration 183 took 52s (31.88% Gen, 66.21% Train). Generation: 16s, Training: 34s. Estimated remaining time: 41h 20m 45s. Estimated total time: 44h 2m 35s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 5s, 500 more iterations: 7h 20m 25s.
[2025-11-13 00:51:32,112][__main__][INFO] - Starting iteration 183.
[2025-11-13 00:51:32,615][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:51:32,615][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:51:48,593][__main__][INFO] - Number of regex retries in iteration 183: 0
[2025-11-13 00:51:48,594][__main__][INFO] - agents played in iteration 183 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:51:49,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:51:49,490][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:51:49,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:51:49,540][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:51:49,540][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:51:49,541][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:51:50,175][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:51:50,628][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:51:51,145][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:51:51,645][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:51:52,141][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:51:52,642][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:51:53,137][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:51:53,631][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:51:54,132][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:51:54,637][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:51:55,139][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:51:55,639][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:51:56,142][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:51:56,642][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:51:57,143][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:51:57,653][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:51:58,153][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:51:58,655][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:51:59,164][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:51:59,668][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:52:00,169][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:52:00,667][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:52:01,167][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:52:01,671][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:52:02,171][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:52:02,668][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:52:03,168][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:52:03,667][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:52:04,167][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:52:04,666][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:52:05,163][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:52:05,670][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:52:06,172][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:52:06,673][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:52:07,173][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:52:07,675][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:52:08,178][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:52:08,682][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:52:09,184][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:52:09,688][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:52:10,189][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:52:10,693][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:52:11,197][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:52:11,698][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:52:12,198][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:52:12,698][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:52:13,199][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:52:13,702][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:52:14,204][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:52:14,708][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:52:15,210][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:52:15,711][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:52:16,215][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:52:16,716][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:52:17,215][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:52:17,713][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:52:18,213][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:52:18,714][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:52:19,214][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:52:19,716][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:52:20,216][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:52:20,716][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:52:21,214][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:52:21,712][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:52:22,211][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10787 tokens.
[2025-11-13 00:52:22,931][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 00:52:23,686][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:52:23,687][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:52:23,689][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:52:24,668][__main__][INFO] - Iteration 184 took 52s (30.70% Gen, 67.42% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 39m 58s. Estimated total time: 43h 22m 41s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 45s, 500 more iterations: 7h 13m 46s.
[2025-11-13 00:52:24,670][__main__][INFO] - Starting iteration 184.
[2025-11-13 00:52:25,145][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:52:25,145][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:52:40,795][__main__][INFO] - Number of regex retries in iteration 184: 0
[2025-11-13 00:52:40,795][__main__][INFO] - agents played in iteration 184 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:52:41,634][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:52:41,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:52:41,687][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:52:41,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:52:41,710][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:52:41,711][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:52:42,343][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:52:42,798][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:52:43,302][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:52:43,801][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:52:44,299][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:52:44,798][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:52:45,296][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:52:45,796][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:52:46,296][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:52:46,797][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:52:47,296][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:52:47,794][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:52:48,297][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:52:48,800][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:52:49,303][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:52:49,806][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:52:50,309][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:52:50,808][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:52:51,308][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:52:51,813][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:52:52,313][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:52:52,814][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:52:53,314][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:52:53,814][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:52:54,315][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:52:54,814][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:52:55,316][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:52:55,816][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:52:56,318][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:52:56,819][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:52:57,318][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:52:57,822][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:52:58,328][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:52:58,829][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:52:59,329][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:52:59,830][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:53:00,329][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:53:00,836][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:53:01,335][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:53:01,837][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:53:02,346][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:53:02,849][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:53:03,351][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:53:03,853][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:53:04,355][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:53:04,854][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:53:05,354][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:53:05,856][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:53:06,357][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:53:06,858][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:53:07,378][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:53:07,880][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:53:08,382][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:53:08,886][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:53:09,388][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:53:09,892][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:53:10,391][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:53:10,891][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:53:11,395][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:53:11,896][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:53:12,396][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:53:12,895][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:53:13,397][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:53:13,901][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:53:14,401][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10758 tokens.
[2025-11-13 00:53:15,123][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 00:53:15,889][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:53:15,891][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:53:15,893][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:53:16,821][__main__][INFO] - Iteration 185 took 51s (30.28% Gen, 67.92% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 20m 14s. Estimated total time: 43h 3m 49s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 7s, 500 more iterations: 7h 10m 38s.
[2025-11-13 00:53:16,823][__main__][INFO] - Starting iteration 185.
[2025-11-13 00:53:17,304][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:53:17,306][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:53:21,141][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:53:22,928][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:53:32,891][__main__][INFO] - Number of regex retries in iteration 185: 2
[2025-11-13 00:53:32,892][__main__][INFO] - agents played in iteration 185 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:53:33,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:53:33,724][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:53:33,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:53:33,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:53:33,773][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:53:33,773][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:53:34,399][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:53:34,872][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:53:35,377][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:53:35,873][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:53:36,392][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:53:36,888][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:53:37,387][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:53:37,886][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:53:38,384][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:53:38,896][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:53:39,394][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:53:39,897][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:53:40,397][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:53:40,898][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:53:41,403][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:53:41,904][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:53:42,398][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:53:42,900][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:53:43,401][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:53:43,903][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:53:44,405][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:53:44,907][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:53:45,412][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:53:45,911][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:53:46,410][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:53:46,909][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:53:47,407][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:53:47,910][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:53:48,410][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:53:48,910][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:53:49,413][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:53:49,913][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:53:50,411][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:53:50,912][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:53:51,413][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:53:51,916][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:53:52,417][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:53:52,918][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:53:53,419][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:53:53,923][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:53:54,425][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:53:54,927][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:53:55,429][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:53:55,932][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:53:56,433][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:53:56,937][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:53:57,438][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:53:57,938][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:53:58,441][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:53:58,944][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:53:59,447][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:53:59,948][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:54:00,448][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:54:00,954][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:54:01,454][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:54:01,953][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:54:02,463][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:54:02,962][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:54:03,468][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:54:03,969][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:54:04,471][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:54:04,989][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:54:05,491][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:54:05,991][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:54:06,490][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10776 tokens.
[2025-11-13 00:54:07,155][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 00:54:07,947][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:54:07,949][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:54:07,950][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:54:08,889][__main__][INFO] - Iteration 186 took 51s (30.21% Gen, 67.96% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 14m 50s. Estimated total time: 42h 59m 18s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 58s, 500 more iterations: 7h 9m 53s.
[2025-11-13 00:54:08,892][__main__][INFO] - Starting iteration 186.
[2025-11-13 00:54:09,379][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:54:09,380][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:54:25,953][__main__][INFO] - Number of regex retries in iteration 186: 0
[2025-11-13 00:54:25,954][__main__][INFO] - agents played in iteration 186 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:54:26,793][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:54:26,815][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:54:26,837][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:54:26,859][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:54:26,859][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:54:26,860][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:54:27,487][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:54:27,943][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:54:28,447][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:54:28,948][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:54:29,449][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:54:29,949][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:54:30,444][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:54:30,948][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:54:31,446][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:54:31,946][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:54:32,453][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:54:32,955][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:54:33,459][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:54:33,961][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:54:34,463][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:54:34,966][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:54:35,467][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:54:35,968][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:54:36,481][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:54:36,982][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:54:37,496][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:54:37,996][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:54:38,499][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:54:39,001][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:54:39,499][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:54:40,008][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:54:40,512][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:54:41,015][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:54:41,515][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:54:42,017][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:54:42,519][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:54:43,023][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:54:43,522][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:54:44,026][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:54:44,528][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:54:45,030][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:54:45,534][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:54:46,036][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:54:46,538][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:54:47,038][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:54:47,541][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:54:48,039][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:54:48,539][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:54:49,040][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:54:49,545][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:54:50,047][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:54:50,554][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:54:51,053][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:54:51,551][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:54:52,053][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:54:52,555][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:54:53,052][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:54:53,554][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:54:54,055][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:54:54,555][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:54:55,054][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:54:55,553][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:54:56,055][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:54:56,555][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:54:57,054][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:54:57,552][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:54:58,049][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:54:58,549][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:54:59,048][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:54:59,545][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10746 tokens.
[2025-11-13 00:55:00,220][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 00:55:01,012][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:55:01,014][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:55:01,015][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:55:02,145][__main__][INFO] - Iteration 187 took 52s (31.41% Gen, 66.45% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 12m 58s. Estimated total time: 43h 58m 19s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 56s, 500 more iterations: 7h 19m 43s.
[2025-11-13 00:55:02,147][__main__][INFO] - Starting iteration 187.
[2025-11-13 00:55:02,619][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:55:02,620][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:55:18,652][__main__][INFO] - Number of regex retries in iteration 187: 0
[2025-11-13 00:55:18,653][__main__][INFO] - agents played in iteration 187 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:55:19,482][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:55:19,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:55:19,535][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:55:19,558][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:55:19,559][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:55:19,560][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:55:20,199][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:55:20,652][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:55:21,152][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:55:21,647][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:55:22,146][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:55:22,641][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:55:23,135][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:55:23,633][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:55:24,132][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:55:24,633][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:55:25,137][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:55:25,637][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:55:26,137][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:55:26,639][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:55:27,142][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:55:27,646][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:55:28,149][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:55:28,649][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:55:29,152][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:55:29,650][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:55:30,153][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:55:30,652][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:55:31,152][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:55:31,659][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:55:32,157][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:55:32,686][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:55:33,192][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:55:33,694][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:55:34,199][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:55:34,707][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:55:35,211][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:55:35,713][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:55:36,215][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:55:36,719][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:55:37,216][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:55:37,718][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:55:38,225][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:55:38,725][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:55:39,224][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:55:39,724][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:55:40,222][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:55:40,729][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:55:41,229][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:55:41,730][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:55:42,229][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:55:42,729][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:55:43,236][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:55:43,738][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:55:44,239][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:55:44,740][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:55:45,234][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:55:45,733][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:55:46,228][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:55:46,724][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:55:47,224][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:55:47,724][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:55:48,226][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:55:48,727][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:55:49,229][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:55:49,728][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:55:50,225][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:55:50,720][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:55:51,220][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:55:51,719][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:55:52,216][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10659 tokens.
[2025-11-13 00:55:52,892][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 00:55:53,682][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:55:53,684][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:55:53,685][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:55:54,661][__main__][INFO] - Iteration 188 took 52s (30.81% Gen, 67.32% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 35m 53s. Estimated total time: 43h 22m 6s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 44s, 500 more iterations: 7h 13m 41s.
[2025-11-13 00:55:54,664][__main__][INFO] - Starting iteration 188.
[2025-11-13 00:55:55,154][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:55:55,155][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:56:06,076][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:56:10,828][__main__][INFO] - Number of regex retries in iteration 188: 1
[2025-11-13 00:56:10,829][__main__][INFO] - agents played in iteration 188 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:56:11,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:56:11,627][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:56:11,653][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:56:11,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:56:11,677][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:56:11,677][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:56:12,314][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:56:12,775][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:56:13,286][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:56:13,787][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:56:14,288][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:56:14,790][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:56:15,291][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:56:15,791][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:56:16,292][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:56:16,789][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:56:17,308][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:56:17,813][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:56:18,314][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:56:18,819][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:56:19,320][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:56:19,824][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:56:20,328][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:56:20,831][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:56:21,336][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:56:21,835][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:56:22,337][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:56:22,836][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:56:23,338][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:56:23,841][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:56:24,340][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:56:24,841][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:56:25,343][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:56:25,843][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:56:26,342][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:56:26,842][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:56:27,346][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:56:27,850][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:56:28,351][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:56:28,849][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:56:29,353][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:56:29,853][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:56:30,358][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:56:30,859][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:56:31,361][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:56:31,865][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:56:32,365][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:56:32,863][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:56:33,362][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:56:33,859][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:56:34,357][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:56:34,854][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:56:35,356][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:56:35,854][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:56:36,353][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:56:36,852][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:56:37,352][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:56:37,852][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:56:38,353][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:56:38,851][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:56:39,349][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:56:39,851][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:56:40,347][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:56:40,846][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:56:41,348][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:56:41,847][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:56:42,348][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:56:42,843][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:56:43,342][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:56:43,841][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:56:44,344][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10746 tokens.
[2025-11-13 00:56:45,017][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.58%, ΔTime: 00:00:32
[2025-11-13 00:56:45,796][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:56:45,798][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:56:45,800][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:56:46,795][__main__][INFO] - Iteration 189 took 51s (30.35% Gen, 67.72% Train). Generation: 15s, Training: 34s. Estimated remaining time: 40h 15m 2s. Estimated total time: 43h 2m 7s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 4s, 500 more iterations: 7h 10m 21s.
[2025-11-13 00:56:46,798][__main__][INFO] - Starting iteration 189.
[2025-11-13 00:56:47,290][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:56:47,291][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:57:03,517][__main__][INFO] - Number of regex retries in iteration 189: 0
[2025-11-13 00:57:03,517][__main__][INFO] - agents played in iteration 189 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:57:04,357][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:57:04,388][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:57:04,414][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:57:04,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:57:04,438][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:57:04,439][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:57:05,063][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:57:05,521][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:57:06,025][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:57:06,525][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:57:07,029][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:57:07,529][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:57:08,028][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:57:08,527][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:57:09,025][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:57:09,526][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:57:10,025][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:57:10,526][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:57:11,034][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:57:11,532][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:57:12,037][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:57:12,536][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:57:13,038][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:57:13,536][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:57:14,034][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:57:14,535][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:57:15,046][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:57:15,545][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:57:16,045][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:57:16,546][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:57:17,044][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:57:17,547][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:57:18,049][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:57:18,553][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:57:19,057][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:57:19,556][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:57:20,058][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:57:20,562][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:57:21,062][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:57:21,564][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:57:22,068][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:57:22,572][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:57:23,078][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:57:23,580][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:57:24,081][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:57:24,591][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:57:25,093][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:57:25,594][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:57:26,105][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:57:26,607][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:57:27,121][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:57:27,623][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:57:28,125][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:57:28,628][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:57:29,132][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:57:29,639][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:57:30,142][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:57:30,646][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:57:31,151][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:57:31,652][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:57:32,154][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:57:32,657][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:57:33,157][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:57:33,661][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:57:34,162][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:57:34,664][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:57:35,170][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:57:35,673][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:57:36,174][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:57:36,673][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:57:37,172][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10722 tokens.
[2025-11-13 00:57:37,844][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.28%, Current % of VRAM taken: 60.54%, Block Peak % of device VRAM: 62.52%, ΔTime: 00:00:32
[2025-11-13 00:57:38,612][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:57:38,613][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:57:38,615][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:57:39,532][__main__][INFO] - Iteration 190 took 52s (31.06% Gen, 67.18% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 44m 8s. Estimated total time: 43h 32m 6s. Time estimates for 10 more iterations: 8m 42s, 100 more iterations: 1h 27m 4s, 500 more iterations: 7h 15m 21s.
[2025-11-13 00:57:39,534][__main__][INFO] - Starting iteration 190.
[2025-11-13 00:57:40,000][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 00:57:40,000][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:57:43,858][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:57:43,964][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:57:56,599][__main__][INFO] - Number of regex retries in iteration 190: 2
[2025-11-13 00:57:56,600][__main__][INFO] - agents played in iteration 190 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:57:57,449][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:57:57,476][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:57:57,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:57:57,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:57:57,524][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:57:57,525][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:57:58,147][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:57:58,613][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:57:59,114][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:57:59,614][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:58:00,113][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:58:00,609][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:58:01,107][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:58:01,611][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:58:02,109][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:58:02,630][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:58:03,128][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:58:03,631][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:58:04,132][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:58:04,634][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:58:05,138][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:58:05,639][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:58:06,140][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:58:06,638][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:58:07,136][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:58:07,637][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:58:08,136][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:58:08,635][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:58:09,142][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:58:09,640][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:58:10,141][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:58:10,641][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:58:11,139][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:58:11,644][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:58:12,146][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:58:12,647][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:58:13,148][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:58:13,649][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:58:14,149][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:58:14,652][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:58:15,155][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:58:15,659][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:58:16,161][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:58:16,668][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:58:17,174][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:58:17,678][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:58:18,180][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:58:18,684][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:58:19,189][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:58:19,690][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:58:20,190][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:58:20,690][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:58:21,191][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:58:21,692][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:58:22,195][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:58:22,696][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:58:23,195][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:58:23,707][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:58:24,209][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:58:24,722][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:58:25,223][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:58:25,721][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:58:26,235][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:58:26,735][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:58:27,234][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:58:27,732][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:58:28,228][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:58:28,736][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:58:29,231][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:58:29,726][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:58:30,226][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10720 tokens.
[2025-11-13 00:58:30,878][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 00:58:31,668][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:58:31,670][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:58:31,672][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:58:33,638][__main__][INFO] - Iteration 191 took 53s (30.95% Gen, 65.39% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 53m 3s. Estimated total time: 44h 41m 55s. Time estimates for 10 more iterations: 8m 56s, 100 more iterations: 1h 29m 23s, 500 more iterations: 7h 26m 59s.
[2025-11-13 00:58:33,640][__main__][INFO] - Starting iteration 191.
[2025-11-13 00:58:34,123][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 00:58:34,123][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:58:45,292][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:58:50,220][__main__][INFO] - Number of regex retries in iteration 191: 1
[2025-11-13 00:58:50,220][__main__][INFO] - agents played in iteration 191 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:58:50,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:58:51,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:58:51,043][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:58:51,065][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:58:51,066][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:58:51,066][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:58:51,687][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:58:52,144][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:58:52,648][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:58:53,147][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:58:53,647][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:58:54,145][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:58:54,646][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:58:55,148][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:58:55,649][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:58:56,153][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:58:56,654][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:58:57,155][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:58:57,652][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:58:58,152][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:58:58,656][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:58:59,156][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:58:59,654][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:59:00,155][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:59:00,655][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:59:01,156][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:59:01,660][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:59:02,160][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:59:02,662][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:59:03,166][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:59:03,669][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:59:04,170][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:59:04,672][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 00:59:05,173][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 00:59:05,674][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 00:59:06,177][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 00:59:06,678][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 00:59:07,178][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 00:59:07,678][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 00:59:08,177][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 00:59:08,679][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 00:59:09,182][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 00:59:09,688][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 00:59:10,189][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 00:59:10,696][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 00:59:11,199][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 00:59:11,700][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 00:59:12,201][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 00:59:12,704][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 00:59:13,215][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 00:59:13,719][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 00:59:14,223][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 00:59:14,728][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 00:59:15,229][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 00:59:15,732][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 00:59:16,234][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 00:59:16,734][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 00:59:17,248][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 00:59:17,748][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 00:59:18,246][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 00:59:18,755][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 00:59:19,262][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 00:59:19,766][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 00:59:20,269][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 00:59:20,772][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 00:59:21,274][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 00:59:21,775][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 00:59:22,274][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 00:59:22,776][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 00:59:23,275][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 00:59:23,774][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10681 tokens.
[2025-11-13 00:59:24,445][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 00:59:25,246][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 00:59:25,248][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 00:59:25,249][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 00:59:26,181][__main__][INFO] - Iteration 192 took 52s (30.92% Gen, 67.29% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 33m 12s. Estimated total time: 43h 22m 56s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 45s, 500 more iterations: 7h 13m 49s.
[2025-11-13 00:59:26,184][__main__][INFO] - Starting iteration 192.
[2025-11-13 00:59:26,690][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 00:59:26,691][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 00:59:31,332][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 00:59:44,984][__main__][INFO] - Number of regex retries in iteration 192: 1
[2025-11-13 00:59:44,985][__main__][INFO] - agents played in iteration 192 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 00:59:45,821][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:59:45,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:59:45,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:59:45,899][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 00:59:45,900][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 00:59:45,900][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 00:59:46,580][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 00:59:47,041][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 00:59:47,558][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 00:59:48,062][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 00:59:48,569][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 00:59:49,076][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 00:59:49,576][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 00:59:50,079][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 00:59:50,581][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 00:59:51,079][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 00:59:51,585][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 00:59:52,083][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 00:59:52,590][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 00:59:53,095][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 00:59:53,596][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 00:59:54,103][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 00:59:54,601][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 00:59:55,113][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 00:59:55,617][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 00:59:56,119][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 00:59:56,620][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 00:59:57,122][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 00:59:57,623][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 00:59:58,133][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 00:59:58,635][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 00:59:59,135][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 00:59:59,642][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:00:00,144][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:00:00,677][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:00:01,187][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:00:01,692][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:00:02,196][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:00:02,696][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:00:03,199][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:00:03,703][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:00:04,204][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:00:04,710][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:00:05,212][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:00:05,715][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:00:06,222][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:00:06,723][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:00:07,224][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:00:07,722][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:00:08,223][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:00:08,725][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:00:09,224][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:00:09,723][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:00:10,231][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:00:10,732][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:00:11,257][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:00:11,752][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:00:12,249][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:00:12,751][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:00:13,249][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:00:13,758][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:00:14,255][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:00:14,751][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:00:15,268][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:00:15,763][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:00:16,260][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:00:16,759][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:00:17,254][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:00:17,760][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:00:18,254][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:00:18,751][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10675 tokens.
[2025-11-13 01:00:19,419][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 01:00:20,182][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:00:20,184][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:00:20,186][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:00:21,174][__main__][INFO] - Iteration 193 took 54s (33.57% Gen, 64.61% Train). Generation: 18s, Training: 35s. Estimated remaining time: 42h 33m 34s. Estimated total time: 45h 24m 14s. Time estimates for 10 more iterations: 9m 4s, 100 more iterations: 1h 30m 48s, 500 more iterations: 7h 34m 2s.
[2025-11-13 01:00:21,176][__main__][INFO] - Starting iteration 193.
[2025-11-13 01:00:21,661][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 01:00:21,662][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:00:29,289][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:00:31,486][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:00:38,817][__main__][INFO] - Number of regex retries in iteration 193: 2
[2025-11-13 01:00:38,818][__main__][INFO] - agents played in iteration 193 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:00:39,694][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:00:39,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:00:39,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:00:39,778][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:00:39,779][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:00:39,780][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:00:40,474][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:00:40,931][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:00:41,474][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:00:41,976][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:00:42,477][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:00:42,992][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:00:43,493][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:00:43,998][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:00:44,500][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:00:45,005][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:00:45,509][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:00:46,012][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:00:46,514][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:00:47,018][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:00:47,520][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:00:48,023][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:00:48,528][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:00:49,030][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:00:49,534][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:00:50,037][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:00:50,539][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:00:51,042][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:00:51,543][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:00:52,041][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:00:52,546][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:00:53,047][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:00:53,546][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:00:54,051][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:00:54,552][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:00:55,057][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:00:55,560][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:00:56,078][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:00:56,580][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:00:57,081][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:00:57,582][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:00:58,086][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:00:58,592][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:00:59,096][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:00:59,597][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:01:00,096][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:01:00,596][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:01:01,098][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:01:01,599][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:01:02,102][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:01:02,606][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:01:03,102][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:01:03,600][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:01:04,101][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:01:04,597][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:01:05,096][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:01:05,593][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:01:06,091][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:01:06,596][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:01:07,092][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:01:07,589][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:01:08,087][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:01:08,583][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:01:09,087][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:01:09,585][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:01:10,083][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:01:10,579][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:01:11,077][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:01:11,575][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:01:12,073][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:01:12,570][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10658 tokens.
[2025-11-13 01:01:13,235][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 01:01:13,996][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:01:13,997][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:01:13,999][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:01:14,941][__main__][INFO] - Iteration 194 took 53s (32.20% Gen, 66.03% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 32m 28s. Estimated total time: 44h 24m 1s. Time estimates for 10 more iterations: 8m 52s, 100 more iterations: 1h 28m 48s, 500 more iterations: 7h 24m 0s.
[2025-11-13 01:01:14,943][__main__][INFO] - Starting iteration 194.
[2025-11-13 01:01:15,415][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 01:01:15,416][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:01:18,591][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:01:30,817][__main__][INFO] - Number of regex retries in iteration 194: 1
[2025-11-13 01:01:30,818][__main__][INFO] - agents played in iteration 194 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:01:31,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:01:31,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:01:31,683][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:01:31,707][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:01:31,707][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:01:31,708][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:01:32,414][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:01:32,873][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:01:33,385][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:01:33,887][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:01:34,398][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:01:34,901][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:01:35,401][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:01:35,907][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:01:36,410][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:01:36,911][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:01:37,413][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:01:37,913][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:01:38,417][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:01:38,916][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:01:39,418][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:01:39,923][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:01:40,422][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:01:40,920][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:01:41,421][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:01:41,924][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:01:42,426][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:01:42,924][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:01:43,426][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:01:43,926][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:01:44,428][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:01:44,933][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:01:45,432][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:01:45,932][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:01:46,435][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:01:46,937][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:01:47,435][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:01:47,935][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:01:48,436][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:01:48,938][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:01:49,438][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:01:49,941][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:01:50,442][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:01:50,940][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:01:51,439][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:01:51,943][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:01:52,446][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:01:52,949][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:01:53,447][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:01:53,947][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:01:54,445][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:01:54,948][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:01:55,448][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:01:55,949][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:01:56,450][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:01:56,950][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:01:57,449][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:01:57,949][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:01:58,448][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:01:58,947][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:01:59,447][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:01:59,946][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:02:00,443][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:02:00,940][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:02:01,432][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:02:01,928][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:02:02,425][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:02:02,922][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:02:03,425][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:02:03,919][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:02:04,417][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10674 tokens.
[2025-11-13 01:02:05,077][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 01:02:05,843][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:02:05,845][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:02:05,847][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:02:07,058][__main__][INFO] - Iteration 195 took 51s (29.82% Gen, 67.83% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 9m 45s. Estimated total time: 43h 2m 10s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 4s, 500 more iterations: 7h 10m 21s.
[2025-11-13 01:02:07,060][__main__][INFO] - Starting iteration 195.
[2025-11-13 01:02:07,547][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 01:02:07,548][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:02:13,513][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:02:23,850][__main__][INFO] - Number of regex retries in iteration 195: 1
[2025-11-13 01:02:23,851][__main__][INFO] - agents played in iteration 195 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:02:24,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:02:24,678][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:02:24,700][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:02:24,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:02:24,723][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:02:24,723][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:02:25,454][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:02:25,916][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:02:26,426][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:02:26,931][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:02:27,436][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:02:27,940][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:02:28,441][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:02:28,944][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:02:29,457][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:02:29,958][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:02:30,457][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:02:30,958][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:02:31,461][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:02:31,963][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:02:32,463][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:02:32,963][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:02:33,462][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:02:33,960][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:02:34,462][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:02:34,963][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:02:35,464][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:02:35,965][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:02:36,469][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:02:36,969][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:02:37,467][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:02:37,968][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:02:38,469][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:02:38,973][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:02:39,476][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:02:39,977][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:02:40,495][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:02:40,992][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:02:41,491][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:02:41,993][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:02:42,498][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:02:43,001][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:02:43,506][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:02:44,010][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:02:44,512][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:02:45,009][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:02:45,511][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:02:46,008][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:02:46,505][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:02:47,013][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:02:47,510][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:02:48,007][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:02:48,505][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:02:49,001][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:02:49,507][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:02:50,003][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:02:50,499][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:02:51,011][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:02:51,509][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:02:52,006][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:02:52,504][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:02:53,004][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:02:53,512][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:02:54,009][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:02:54,506][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:02:55,001][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:02:55,497][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:02:56,004][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:02:56,501][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:02:56,996][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:02:57,504][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10685 tokens.
[2025-11-13 01:02:58,153][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 01:02:58,952][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:02:58,953][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:02:58,955][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:02:59,934][__main__][INFO] - Iteration 196 took 52s (31.12% Gen, 67.01% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 46m 2s. Estimated total time: 43h 39m 20s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 18s, 500 more iterations: 7h 16m 33s.
[2025-11-13 01:02:59,936][__main__][INFO] - Starting iteration 196.
[2025-11-13 01:03:00,439][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 01:03:00,440][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:03:07,175][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:03:17,454][__main__][INFO] - Number of regex retries in iteration 196: 1
[2025-11-13 01:03:17,455][__main__][INFO] - agents played in iteration 196 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:03:18,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:03:18,343][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:03:18,366][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:03:18,388][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:03:18,388][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:03:18,389][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:03:19,085][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:03:19,558][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:03:20,079][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:03:20,583][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:03:21,091][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:03:21,597][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:03:22,103][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:03:22,621][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:03:23,124][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:03:23,625][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:03:24,129][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:03:24,631][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:03:25,136][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:03:25,637][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:03:26,138][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:03:26,639][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:03:27,139][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:03:27,640][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:03:28,137][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:03:28,635][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:03:29,139][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:03:29,640][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:03:30,141][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:03:30,641][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:03:31,143][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:03:31,647][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:03:32,149][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:03:32,651][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:03:33,153][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:03:33,652][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:03:34,154][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:03:34,661][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:03:35,164][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:03:35,670][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:03:36,173][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:03:36,673][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:03:37,172][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:03:37,670][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:03:38,169][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:03:38,669][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:03:39,168][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:03:39,667][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:03:40,167][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:03:40,665][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:03:41,166][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:03:41,665][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:03:42,164][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:03:42,660][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:03:43,158][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:03:43,657][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:03:44,155][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:03:44,650][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:03:45,150][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:03:45,649][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:03:46,147][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:03:46,645][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:03:47,142][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:03:47,642][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:03:48,137][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:03:48,632][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:03:49,129][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:03:49,623][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:03:50,125][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:03:50,620][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:03:51,119][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10729 tokens.
[2025-11-13 01:03:51,798][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 01:03:52,583][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:03:52,585][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:03:52,587][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:03:53,536][__main__][INFO] - Iteration 197 took 53s (32.04% Gen, 66.16% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 20m 40s. Estimated total time: 44h 14m 52s. Time estimates for 10 more iterations: 8m 50s, 100 more iterations: 1h 28m 29s, 500 more iterations: 7h 22m 28s.
[2025-11-13 01:03:53,538][__main__][INFO] - Starting iteration 197.
[2025-11-13 01:03:54,007][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 01:03:54,008][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:03:57,915][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:04:10,881][__main__][INFO] - Number of regex retries in iteration 197: 1
[2025-11-13 01:04:10,881][__main__][INFO] - agents played in iteration 197 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:04:11,687][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:04:11,709][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:04:11,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:04:11,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:04:11,754][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:04:11,755][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:04:12,477][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:04:12,932][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:04:13,437][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:04:13,939][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:04:14,438][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:04:14,945][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:04:15,443][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:04:15,940][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:04:16,438][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:04:16,941][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:04:17,443][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:04:17,947][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:04:18,450][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:04:18,947][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:04:19,448][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:04:19,948][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:04:20,450][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:04:20,949][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:04:21,449][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:04:21,949][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:04:22,450][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:04:22,954][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:04:23,453][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:04:23,954][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:04:24,455][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:04:24,955][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:04:25,460][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:04:25,961][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:04:26,457][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:04:26,952][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:04:27,451][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:04:27,952][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:04:28,452][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:04:28,971][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:04:29,475][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:04:29,972][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:04:30,474][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:04:30,973][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:04:31,473][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:04:31,976][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:04:32,474][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:04:32,972][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:04:33,471][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:04:33,969][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:04:34,470][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:04:34,967][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:04:35,464][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:04:35,965][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:04:36,463][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:04:36,961][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:04:37,461][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:04:37,957][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:04:38,455][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:04:38,954][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:04:39,454][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:04:39,963][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:04:40,461][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:04:40,959][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:04:41,458][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:04:41,955][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:04:42,453][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:04:42,952][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:04:43,449][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:04:43,958][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:04:44,456][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10705 tokens.
[2025-11-13 01:04:45,137][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 01:04:45,915][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:04:45,917][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:04:45,919][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:04:46,946][__main__][INFO] - Iteration 198 took 52s (31.87% Gen, 66.18% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 11m 53s. Estimated total time: 44h 6m 59s. Time estimates for 10 more iterations: 8m 49s, 100 more iterations: 1h 28m 13s, 500 more iterations: 7h 21m 9s.
[2025-11-13 01:04:46,949][__main__][INFO] - Starting iteration 198.
[2025-11-13 01:04:47,431][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 01:04:47,432][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:05:04,467][__main__][INFO] - Number of regex retries in iteration 198: 0
[2025-11-13 01:05:04,468][__main__][INFO] - agents played in iteration 198 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:05:05,300][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:05:05,322][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:05:05,344][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:05:05,366][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:05:05,367][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:05:05,368][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:05:06,054][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:05:06,510][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:05:07,015][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:05:07,518][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:05:08,017][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:05:08,515][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:05:09,018][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:05:09,517][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:05:10,016][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:05:10,513][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:05:11,012][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:05:11,516][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:05:12,017][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:05:12,514][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:05:13,013][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:05:13,513][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:05:14,018][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:05:14,521][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:05:15,021][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:05:15,534][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:05:16,034][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:05:16,543][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:05:17,046][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:05:17,545][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:05:18,045][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:05:18,545][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:05:19,044][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:05:19,548][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:05:20,051][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:05:20,560][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:05:21,061][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:05:21,561][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:05:22,062][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:05:22,561][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:05:23,058][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:05:23,556][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:05:24,054][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:05:24,560][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:05:25,058][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:05:25,554][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:05:26,055][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:05:26,552][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:05:27,050][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:05:27,548][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:05:28,044][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:05:28,562][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:05:29,059][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:05:29,553][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:05:30,053][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:05:30,551][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:05:31,048][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:05:31,543][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:05:32,038][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:05:32,545][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:05:33,042][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:05:33,538][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:05:34,044][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:05:34,541][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:05:35,046][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:05:35,543][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:05:36,040][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:05:36,551][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:05:37,048][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:05:37,543][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:05:38,047][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10604 tokens.
[2025-11-13 01:05:38,705][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:32
[2025-11-13 01:05:39,489][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:05:39,490][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:05:39,492][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:05:40,497][__main__][INFO] - Iteration 199 took 53s (32.10% Gen, 66.00% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 17m 22s. Estimated total time: 44h 13m 21s. Time estimates for 10 more iterations: 8m 50s, 100 more iterations: 1h 28m 26s, 500 more iterations: 7h 22m 13s.
[2025-11-13 01:05:40,499][__main__][INFO] - Starting iteration 199.
[2025-11-13 01:05:40,986][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 01:05:40,986][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:05:59,833][__main__][INFO] - Number of regex retries in iteration 199: 0
[2025-11-13 01:05:59,834][__main__][INFO] - agents played in iteration 199 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:06:00,680][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:06:00,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:06:00,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:06:00,748][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:06:00,749][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:06:00,750][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:06:01,441][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:06:01,894][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:06:02,406][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:06:02,916][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:06:03,416][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:06:03,920][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:06:04,426][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:06:04,928][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:06:05,429][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:06:05,925][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:06:06,422][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:06:06,922][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:06:07,421][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:06:07,919][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:06:08,419][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:06:08,915][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:06:09,415][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:06:09,911][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:06:10,409][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:06:10,910][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:06:11,406][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:06:11,907][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:06:12,410][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:06:12,909][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:06:13,410][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:06:13,912][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:06:14,411][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:06:14,912][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:06:15,414][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:06:15,913][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:06:16,409][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:06:16,907][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:06:17,405][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:06:17,903][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:06:18,400][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:06:18,901][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:06:19,404][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:06:19,903][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:06:20,401][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:06:20,899][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:06:21,399][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:06:21,896][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:06:22,394][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:06:22,890][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:06:23,385][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:06:23,890][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:06:24,386][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:06:24,880][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:06:25,379][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:06:25,875][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:06:26,373][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:06:26,870][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:06:27,369][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:06:27,884][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:06:28,381][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:06:28,880][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:06:29,392][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:06:29,892][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:06:30,392][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:06:30,890][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:06:31,388][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:06:31,893][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:06:32,392][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:06:32,892][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:06:33,395][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10605 tokens.
[2025-11-13 01:06:34,044][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 01:06:34,830][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:06:34,832][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:06:34,833][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:06:35,773][__main__][INFO] - Iteration 200 took 54s (34.40% Gen, 63.88% Train). Generation: 18s, Training: 34s. Estimated remaining time: 42h 42m 30s. Estimated total time: 45h 39m 24s. Time estimates for 10 more iterations: 9m 7s, 100 more iterations: 1h 31m 18s, 500 more iterations: 7h 36m 34s.
[2025-11-13 01:06:35,775][__main__][INFO] - Starting iteration 200.
[2025-11-13 01:06:36,268][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 01:06:36,268][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:06:52,485][__main__][INFO] - Number of regex retries in iteration 200: 0
[2025-11-13 01:06:52,486][__main__][INFO] - agents played in iteration 200 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:06:53,327][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:06:53,354][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:06:53,378][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:06:53,401][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:06:53,401][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:06:53,402][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:06:54,091][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:06:54,548][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:06:55,054][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:06:55,553][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:06:56,051][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:06:56,555][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:06:57,070][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:06:57,573][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:06:58,095][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:06:58,597][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:06:59,099][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:06:59,613][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:07:00,114][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:07:00,616][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:07:01,113][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:07:01,613][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:07:02,114][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:07:02,614][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:07:03,113][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:07:03,615][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:07:04,113][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:07:04,613][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:07:05,113][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:07:05,614][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:07:06,114][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:07:06,616][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:07:07,117][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:07:07,619][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:07:08,120][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:07:08,626][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:07:09,126][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:07:09,626][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:07:10,126][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:07:10,621][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:07:11,117][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:07:11,614][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:07:12,110][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:07:12,611][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:07:13,107][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:07:13,607][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:07:14,105][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:07:14,604][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:07:15,103][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:07:15,600][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:07:16,101][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:07:16,606][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:07:17,104][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:07:17,601][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:07:18,104][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:07:18,600][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:07:19,096][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:07:19,593][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:07:20,091][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:07:20,603][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:07:21,097][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:07:21,594][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:07:22,098][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:07:22,596][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:07:23,098][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:07:23,596][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:07:24,094][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:07:24,607][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:07:25,105][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:07:25,603][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:07:26,101][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10653 tokens.
[2025-11-13 01:07:26,756][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 01:07:27,547][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:07:27,548][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:07:27,550][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:07:29,324][__main__][INFO] - Iteration 201 took 53s (30.57% Gen, 66.09% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 15m 4s. Estimated total time: 44h 12m 51s. Time estimates for 10 more iterations: 8m 50s, 100 more iterations: 1h 28m 25s, 500 more iterations: 7h 22m 8s.
[2025-11-13 01:07:29,327][__main__][INFO] - Starting iteration 201.
[2025-11-13 01:07:29,804][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:07:29,805][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:07:47,021][__main__][INFO] - Number of regex retries in iteration 201: 0
[2025-11-13 01:07:47,021][__main__][INFO] - agents played in iteration 201 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:07:47,866][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:07:47,889][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:07:47,921][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:07:47,950][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:07:47,951][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:07:47,952][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:07:48,674][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:07:49,133][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:07:49,640][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:07:50,143][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:07:50,643][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:07:51,139][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:07:51,635][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:07:52,133][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:07:52,632][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:07:53,133][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:07:53,629][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:07:54,128][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:07:54,629][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:07:55,126][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:07:55,626][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:07:56,124][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:07:56,621][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:07:57,122][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:07:57,620][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:07:58,120][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:07:58,627][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:07:59,124][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:07:59,623][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:08:00,122][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:08:00,621][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:08:01,122][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:08:01,619][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:08:02,118][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:08:02,622][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:08:03,118][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:08:03,614][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:08:04,112][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:08:04,608][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:08:05,114][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:08:05,609][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:08:06,104][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:08:06,615][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:08:07,110][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:08:07,604][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:08:08,101][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:08:08,595][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:08:09,095][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:08:09,597][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:08:10,095][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:08:10,594][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:08:11,091][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:08:11,590][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:08:12,090][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:08:12,588][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:08:13,084][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:08:13,589][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:08:14,090][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:08:14,596][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:08:15,090][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:08:15,585][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:08:16,090][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:08:16,588][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:08:17,086][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:08:17,583][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:08:18,079][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:08:18,577][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:08:19,073][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:08:19,571][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:08:20,070][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:08:20,565][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10610 tokens.
[2025-11-13 01:08:21,242][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 01:08:22,021][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:08:22,022][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:08:22,024][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:08:23,133][__main__][INFO] - Iteration 202 took 53s (32.28% Gen, 65.63% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 27m 49s. Estimated total time: 44h 26m 31s. Time estimates for 10 more iterations: 8m 53s, 100 more iterations: 1h 28m 53s, 500 more iterations: 7h 24m 25s.
[2025-11-13 01:08:23,136][__main__][INFO] - Starting iteration 202.
[2025-11-13 01:08:23,629][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:08:23,629][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:08:37,961][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:08:42,142][__main__][INFO] - Number of regex retries in iteration 202: 1
[2025-11-13 01:08:42,143][__main__][INFO] - agents played in iteration 202 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:08:43,025][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:08:43,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:08:43,081][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:08:43,104][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:08:43,104][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:08:43,106][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:08:43,760][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:08:44,216][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:08:44,724][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:08:45,222][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:08:45,718][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:08:46,232][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:08:46,731][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:08:47,228][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:08:47,727][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:08:48,227][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:08:48,732][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:08:49,231][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:08:49,731][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:08:50,241][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:08:50,740][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:08:51,239][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:08:51,743][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:08:52,242][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:08:52,741][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:08:53,242][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:08:53,741][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:08:54,252][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:08:54,756][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:08:55,258][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:08:55,753][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:08:56,252][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:08:56,753][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:08:57,251][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:08:57,749][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:08:58,249][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:08:58,747][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:08:59,248][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:08:59,749][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:09:00,246][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:09:00,748][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:09:01,248][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:09:01,747][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:09:02,245][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:09:02,742][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:09:03,239][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:09:03,733][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:09:04,231][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:09:04,741][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:09:05,236][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:09:05,730][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:09:06,227][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:09:06,725][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:09:07,230][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:09:07,723][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:09:08,221][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:09:08,718][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:09:09,213][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:09:09,710][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:09:10,204][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:09:10,700][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:09:11,206][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:09:11,701][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:09:12,197][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:09:12,718][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:09:13,215][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:09:13,715][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:09:14,218][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:09:14,720][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:09:15,239][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:09:15,741][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10581 tokens.
[2025-11-13 01:09:16,472][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 01:09:17,203][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:09:17,204][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:09:17,205][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:09:18,141][__main__][INFO] - Iteration 203 took 54s (33.96% Gen, 64.32% Train). Generation: 18s, Training: 35s. Estimated remaining time: 42h 26m 2s. Estimated total time: 45h 25m 38s. Time estimates for 10 more iterations: 9m 5s, 100 more iterations: 1h 30m 51s, 500 more iterations: 7h 34m 16s.
[2025-11-13 01:09:18,143][__main__][INFO] - Starting iteration 203.
[2025-11-13 01:09:18,627][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:09:18,628][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:09:27,695][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:09:31,892][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:09:35,567][__main__][INFO] - Number of regex retries in iteration 203: 2
[2025-11-13 01:09:35,568][__main__][INFO] - agents played in iteration 203 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:09:36,525][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:09:36,549][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:09:36,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:09:36,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:09:36,597][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:09:36,598][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:09:37,244][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:09:37,704][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:09:38,207][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:09:38,710][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:09:39,218][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:09:39,720][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:09:40,235][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:09:40,740][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:09:41,243][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:09:41,738][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:09:42,238][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:09:42,741][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:09:43,239][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:09:43,738][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:09:44,246][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:09:44,746][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:09:45,249][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:09:45,744][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:09:46,240][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:09:46,745][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:09:47,244][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:09:47,744][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:09:48,250][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:09:48,747][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:09:49,245][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:09:49,743][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:09:50,239][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:09:50,742][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:09:51,237][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:09:51,734][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:09:52,233][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:09:52,730][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:09:53,229][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:09:53,729][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:09:54,228][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:09:54,742][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:09:55,240][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:09:55,735][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:09:56,232][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:09:56,731][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:09:57,230][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:09:57,728][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:09:58,227][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:09:58,742][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:09:59,239][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:09:59,736][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:10:00,235][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:10:00,732][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:10:01,227][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:10:01,729][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:10:02,228][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:10:02,741][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:10:03,238][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:10:03,735][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:10:04,233][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:10:04,728][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:10:05,230][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:10:05,730][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:10:06,231][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:10:06,736][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:10:07,239][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:10:07,749][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:10:08,253][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:10:08,763][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:10:09,266][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10679 tokens.
[2025-11-13 01:10:09,987][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 01:10:10,742][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:10:10,744][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:10:10,745][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:10:11,830][__main__][INFO] - Iteration 204 took 53s (31.84% Gen, 66.12% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 19m 38s. Estimated total time: 44h 20m 9s. Time estimates for 10 more iterations: 8m 52s, 100 more iterations: 1h 28m 40s, 500 more iterations: 7h 23m 21s.
[2025-11-13 01:10:11,832][__main__][INFO] - Starting iteration 204.
[2025-11-13 01:10:12,306][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:10:12,307][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:10:16,356][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 30 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:10:27,716][__main__][INFO] - Number of regex retries in iteration 204: 1
[2025-11-13 01:10:27,717][__main__][INFO] - agents played in iteration 204 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:10:28,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:10:28,521][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:10:28,544][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:10:28,580][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:10:28,580][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:10:28,581][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:10:29,226][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:10:29,683][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:10:30,190][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:10:30,693][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:10:31,196][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:10:31,695][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:10:32,196][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:10:32,695][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:10:33,196][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:10:33,694][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:10:34,200][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:10:34,697][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:10:35,195][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:10:35,697][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:10:36,194][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:10:36,695][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:10:37,197][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:10:37,696][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:10:38,206][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:10:38,709][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:10:39,212][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:10:39,714][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:10:40,215][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:10:40,715][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:10:41,214][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:10:41,712][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:10:42,212][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:10:42,711][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:10:43,227][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:10:43,727][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:10:44,223][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:10:44,725][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:10:45,223][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:10:45,721][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:10:46,221][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:10:46,719][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:10:47,232][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:10:47,732][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:10:48,231][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:10:48,731][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:10:49,230][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:10:49,740][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:10:50,239][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:10:50,738][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:10:51,239][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:10:51,737][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:10:52,237][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:10:52,734][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:10:53,232][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:10:53,745][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:10:54,244][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:10:54,739][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:10:55,241][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:10:55,740][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:10:56,241][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:10:56,746][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:10:57,255][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:10:57,785][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:10:58,318][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:10:58,832][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:10:59,336][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:10:59,836][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:11:00,339][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:11:00,843][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:11:01,343][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10665 tokens.
[2025-11-13 01:11:02,089][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 01:11:02,840][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:11:02,842][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:11:02,844][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:11:03,739][__main__][INFO] - Iteration 205 took 51s (29.96% Gen, 68.30% Train). Generation: 15s, Training: 35s. Estimated remaining time: 39h 50m 18s. Estimated total time: 42h 51m 40s. Time estimates for 10 more iterations: 8m 34s, 100 more iterations: 1h 25m 43s, 500 more iterations: 7h 8m 36s.
[2025-11-13 01:11:03,741][__main__][INFO] - Starting iteration 205.
[2025-11-13 01:11:04,222][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:11:04,223][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:11:22,221][__main__][INFO] - Number of regex retries in iteration 205: 0
[2025-11-13 01:11:22,221][__main__][INFO] - agents played in iteration 205 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:11:23,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:11:23,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:11:23,065][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:11:23,088][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:11:23,088][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:11:23,089][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:11:23,736][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:11:24,195][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:11:24,696][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:11:25,200][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:11:25,696][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:11:26,196][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:11:26,695][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:11:27,196][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:11:27,695][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:11:28,195][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:11:28,701][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:11:29,219][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:11:29,718][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:11:30,218][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:11:30,730][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:11:31,230][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:11:31,730][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:11:32,238][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:11:32,737][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:11:33,248][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:11:33,746][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:11:34,243][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:11:34,742][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:11:35,242][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:11:35,865][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:11:36,374][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:11:36,870][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:11:37,370][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:11:37,871][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:11:38,369][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:11:38,864][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:11:39,362][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:11:39,868][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:11:40,368][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:11:40,870][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:11:41,367][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:11:41,865][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:11:42,368][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:11:42,867][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:11:43,365][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:11:43,866][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:11:44,365][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:11:44,862][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:11:45,359][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:11:45,856][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:11:46,359][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:11:46,859][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:11:47,364][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:11:47,864][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:11:48,367][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:11:48,873][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:11:49,375][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:11:49,880][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:11:50,385][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:11:50,886][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:11:51,386][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:11:51,889][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:11:52,392][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:11:52,895][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:11:53,396][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:11:53,894][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:11:54,398][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:11:54,897][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:11:55,400][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:11:55,901][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10559 tokens.
[2025-11-13 01:11:56,577][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 01:11:57,352][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:11:57,354][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:11:57,357][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:11:58,283][__main__][INFO] - Iteration 206 took 54s (33.29% Gen, 64.99% Train). Generation: 17s, Training: 35s. Estimated remaining time: 42h 0m 49s. Estimated total time: 45h 3m 5s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 6s, 500 more iterations: 7h 30m 30s.
[2025-11-13 01:11:58,285][__main__][INFO] - Starting iteration 206.
[2025-11-13 01:11:58,782][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:11:58,782][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:12:03,518][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:12:16,886][__main__][INFO] - Number of regex retries in iteration 206: 1
[2025-11-13 01:12:16,886][__main__][INFO] - agents played in iteration 206 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:12:17,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:12:17,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:12:17,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:12:17,829][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:12:17,829][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:12:17,830][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:12:18,473][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:12:18,929][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:12:19,449][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:12:19,949][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:12:20,460][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:12:20,957][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:12:21,457][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:12:21,959][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:12:22,464][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:12:22,965][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:12:23,463][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:12:23,959][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:12:24,474][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:12:24,975][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:12:25,472][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:12:25,969][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:12:26,466][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:12:26,965][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:12:27,464][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:12:27,962][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:12:28,478][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:12:28,980][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:12:29,480][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:12:29,977][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:12:30,475][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:12:30,979][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:12:31,478][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:12:31,976][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:12:32,484][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:12:32,982][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:12:33,485][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:12:33,988][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:12:34,488][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:12:34,990][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:12:35,490][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:12:35,988][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:12:36,491][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:12:36,988][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:12:37,491][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:12:37,989][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:12:38,489][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:12:38,993][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:12:39,490][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:12:39,990][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:12:40,496][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:12:40,999][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:12:41,504][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:12:42,007][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:12:42,512][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:12:43,016][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:12:43,519][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:12:44,025][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:12:44,526][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:12:45,027][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:12:45,529][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:12:46,030][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:12:46,533][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:12:47,035][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:12:47,537][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:12:48,039][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:12:48,541][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:12:49,042][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:12:49,556][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:12:50,055][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:12:50,558][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10576 tokens.
[2025-11-13 01:12:51,261][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.37%, ΔTime: 00:00:32
[2025-11-13 01:12:52,031][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:12:52,033][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:12:52,035][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:12:53,018][__main__][INFO] - Iteration 207 took 54s (33.38% Gen, 64.81% Train). Generation: 18s, Training: 35s. Estimated remaining time: 42h 8m 40s. Estimated total time: 45h 11m 52s. Time estimates for 10 more iterations: 9m 2s, 100 more iterations: 1h 30m 23s, 500 more iterations: 7h 31m 58s.
[2025-11-13 01:12:53,021][__main__][INFO] - Starting iteration 207.
[2025-11-13 01:12:53,491][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:12:53,491][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:13:10,975][__main__][INFO] - Number of regex retries in iteration 207: 0
[2025-11-13 01:13:10,976][__main__][INFO] - agents played in iteration 207 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:13:11,858][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:13:11,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:13:11,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:13:11,935][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:13:11,936][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:13:11,937][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:13:12,592][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:13:13,058][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:13:13,564][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:13:14,069][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:13:14,571][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:13:15,074][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:13:15,578][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:13:16,077][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:13:16,573][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:13:17,089][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:13:17,593][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:13:18,091][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:13:18,592][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:13:19,091][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:13:19,593][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:13:20,095][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:13:20,598][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:13:21,100][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:13:21,602][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:13:22,105][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:13:22,599][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:13:23,101][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:13:23,609][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:13:24,109][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:13:24,613][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:13:25,115][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:13:25,613][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:13:26,111][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:13:26,612][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:13:27,114][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:13:27,616][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:13:28,118][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:13:28,620][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:13:29,116][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:13:29,616][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:13:30,117][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:13:30,616][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:13:31,118][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:13:31,622][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:13:32,124][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:13:32,629][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:13:33,133][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:13:33,635][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:13:34,138][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:13:34,640][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:13:35,142][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:13:35,647][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:13:36,149][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:13:36,652][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:13:37,154][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:13:37,653][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:13:38,180][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:13:38,682][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:13:39,183][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:13:39,684][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:13:40,183][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:13:40,684][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:13:41,188][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:13:41,686][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:13:42,200][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:13:42,698][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:13:43,199][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:13:43,698][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:13:44,198][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:13:44,702][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10564 tokens.
[2025-11-13 01:13:45,382][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 01:13:46,166][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:13:46,167][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:13:46,169][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:13:47,077][__main__][INFO] - Iteration 208 took 53s (32.63% Gen, 65.68% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 35m 16s. Estimated total time: 44h 39m 22s. Time estimates for 10 more iterations: 8m 55s, 100 more iterations: 1h 29m 18s, 500 more iterations: 7h 26m 33s.
[2025-11-13 01:13:47,080][__main__][INFO] - Starting iteration 208.
[2025-11-13 01:13:47,563][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:13:47,563][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:13:51,260][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:13:51,315][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:13:51,437][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:14:03,331][__main__][INFO] - Number of regex retries in iteration 208: 3
[2025-11-13 01:14:03,332][__main__][INFO] - agents played in iteration 208 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:14:04,154][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:14:04,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:14:04,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:14:04,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:14:04,245][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:14:04,245][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:14:04,900][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:14:05,358][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:14:05,877][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:14:06,378][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:14:06,882][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:14:07,385][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:14:07,886][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:14:08,400][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:14:08,899][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:14:09,395][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:14:09,896][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:14:10,393][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:14:10,898][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:14:11,397][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:14:11,895][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:14:12,395][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:14:12,895][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:14:13,393][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:14:13,894][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:14:14,391][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:14:14,906][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:14:15,404][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:14:15,902][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:14:16,397][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:14:16,895][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:14:17,400][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:14:17,897][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:14:18,395][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:14:18,895][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:14:19,394][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:14:19,894][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:14:20,395][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:14:20,893][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:14:21,407][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:14:21,905][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:14:22,405][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:14:22,905][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:14:23,403][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:14:23,904][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:14:24,406][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:14:24,909][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:14:25,412][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:14:25,912][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:14:26,417][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:14:26,919][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:14:27,420][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:14:27,922][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:14:28,426][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:14:28,930][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:14:29,431][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:14:29,933][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:14:30,438][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:14:30,939][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:14:31,439][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:14:31,941][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:14:32,441][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:14:32,941][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:14:33,442][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:14:33,939][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:14:34,440][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:14:34,940][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:14:35,440][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:14:35,941][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:14:36,442][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:14:36,942][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10605 tokens.
[2025-11-13 01:14:37,631][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 01:14:38,437][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:14:38,438][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:14:38,441][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:14:39,354][__main__][INFO] - Iteration 209 took 51s (30.45% Gen, 67.79% Train). Generation: 15s, Training: 35s. Estimated remaining time: 40h 4m 37s. Estimated total time: 43h 9m 35s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 19s, 500 more iterations: 7h 11m 35s.
[2025-11-13 01:14:39,356][__main__][INFO] - Starting iteration 209.
[2025-11-13 01:14:39,857][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:14:39,858][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:14:52,127][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:14:57,654][__main__][INFO] - Number of regex retries in iteration 209: 1
[2025-11-13 01:14:57,655][__main__][INFO] - agents played in iteration 209 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:14:58,523][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:14:58,546][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:14:58,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:14:58,592][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:14:58,592][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:14:58,593][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:14:59,283][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:14:59,742][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:15:00,247][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:15:00,747][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:15:01,249][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:15:01,750][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:15:02,254][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:15:02,756][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:15:03,255][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:15:03,754][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:15:04,254][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:15:04,753][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:15:05,252][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:15:05,751][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:15:06,250][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:15:06,749][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:15:07,249][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:15:07,750][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:15:08,251][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:15:08,753][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:15:09,252][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:15:09,750][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:15:10,250][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:15:10,750][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:15:11,251][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:15:11,750][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:15:12,249][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:15:12,749][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:15:13,247][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:15:13,747][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:15:14,249][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:15:14,752][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:15:15,255][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:15:15,757][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:15:16,259][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:15:16,760][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:15:17,261][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:15:17,764][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:15:18,265][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:15:18,769][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:15:19,272][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:15:19,774][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:15:20,291][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:15:20,794][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:15:21,295][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:15:21,797][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:15:22,299][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:15:22,804][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:15:23,306][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:15:23,808][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:15:24,310][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:15:24,810][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:15:25,308][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:15:25,803][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:15:26,301][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:15:26,810][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:15:27,309][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:15:27,806][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:15:28,303][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:15:28,800][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:15:29,306][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:15:29,801][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:15:30,297][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:15:30,795][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:15:31,294][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10571 tokens.
[2025-11-13 01:15:31,984][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 01:15:32,747][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:15:32,748][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:15:32,750][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:15:33,695][__main__][INFO] - Iteration 210 took 53s (33.06% Gen, 65.19% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 46m 1s. Estimated total time: 44h 51m 53s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 43s, 500 more iterations: 7h 28m 38s.
[2025-11-13 01:15:33,697][__main__][INFO] - Starting iteration 210.
[2025-11-13 01:15:34,193][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 01:15:34,194][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:15:52,093][__main__][INFO] - Number of regex retries in iteration 210: 0
[2025-11-13 01:15:52,094][__main__][INFO] - agents played in iteration 210 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:15:52,939][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:15:52,966][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:15:52,992][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:15:53,018][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:15:53,020][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:15:53,022][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:15:53,714][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:15:54,169][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:15:54,674][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:15:55,176][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:15:55,675][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:15:56,190][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:15:56,692][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:15:57,208][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:15:57,708][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:15:58,207][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:15:58,722][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:15:59,219][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:15:59,716][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:16:00,215][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:16:00,717][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:16:01,219][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:16:01,716][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:16:02,211][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:16:02,715][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:16:03,212][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:16:03,708][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:16:04,204][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:16:04,699][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:16:05,194][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:16:05,692][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:16:06,188][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:16:06,710][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:16:07,210][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:16:07,711][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:16:08,214][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:16:08,714][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:16:09,214][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:16:09,715][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:16:10,217][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:16:10,724][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:16:11,226][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:16:11,724][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:16:12,232][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:16:12,731][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:16:13,235][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:16:13,736][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:16:14,235][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:16:14,736][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:16:15,235][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:16:15,734][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:16:16,230][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:16:16,730][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:16:17,230][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:16:17,728][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:16:18,226][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:16:18,726][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:16:19,222][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:16:19,721][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:16:20,219][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:16:20,717][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:16:21,226][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:16:21,725][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:16:22,223][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:16:22,723][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:16:23,221][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:16:23,724][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:16:24,221][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:16:24,720][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:16:25,221][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:16:25,721][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10433 tokens.
[2025-11-13 01:16:26,487][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 01:16:27,259][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:16:27,260][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:16:27,262][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:16:29,040][__main__][INFO] - Iteration 211 took 54s (32.64% Gen, 64.12% Train). Generation: 17s, Training: 35s. Estimated remaining time: 42h 35m 32s. Estimated total time: 45h 42m 19s. Time estimates for 10 more iterations: 9m 8s, 100 more iterations: 1h 31m 24s, 500 more iterations: 7h 37m 3s.
[2025-11-13 01:16:29,042][__main__][INFO] - Starting iteration 211.
[2025-11-13 01:16:29,528][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:16:29,529][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:16:46,177][__main__][INFO] - Number of regex retries in iteration 211: 0
[2025-11-13 01:16:46,178][__main__][INFO] - agents played in iteration 211 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:16:47,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:16:47,036][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:16:47,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:16:47,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:16:47,085][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:16:47,086][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:16:47,785][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:16:48,244][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:16:48,745][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:16:49,245][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:16:49,745][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:16:50,243][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:16:50,739][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:16:51,236][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:16:51,737][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:16:52,231][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:16:52,727][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:16:53,232][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:16:53,730][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:16:54,226][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:16:54,725][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:16:55,219][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:16:55,718][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:16:56,215][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:16:56,714][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:16:57,212][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:16:57,712][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:16:58,221][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:16:58,718][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:16:59,220][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:16:59,735][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:17:00,237][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:17:00,737][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:17:01,239][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:17:01,739][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:17:02,242][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:17:02,742][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:17:03,244][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:17:03,745][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:17:04,242][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:17:04,742][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:17:05,241][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:17:05,740][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:17:06,238][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:17:06,737][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:17:07,233][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:17:07,736][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:17:08,238][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:17:08,738][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:17:09,234][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:17:09,736][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:17:10,236][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:17:10,733][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:17:11,233][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:17:11,733][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:17:12,233][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:17:12,731][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:17:13,232][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:17:13,733][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:17:14,239][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:17:14,738][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:17:15,240][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:17:15,736][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:17:16,239][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:17:16,747][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:17:17,246][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:17:17,749][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:17:18,255][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:17:18,757][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:17:19,258][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:17:19,759][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10535 tokens.
[2025-11-13 01:17:20,473][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 01:17:21,255][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:17:21,257][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:17:21,259][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:17:22,241][__main__][INFO] - Iteration 212 took 52s (31.58% Gen, 66.55% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 48m 2s. Estimated total time: 43h 55m 42s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 51s, 500 more iterations: 7h 19m 17s.
[2025-11-13 01:17:22,244][__main__][INFO] - Starting iteration 212.
[2025-11-13 01:17:22,729][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:17:22,730][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:17:40,384][__main__][INFO] - Number of regex retries in iteration 212: 0
[2025-11-13 01:17:40,385][__main__][INFO] - agents played in iteration 212 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:17:41,169][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:17:41,196][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:17:41,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:17:41,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:17:41,244][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:17:41,245][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:17:41,928][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:17:42,385][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:17:42,885][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:17:43,381][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:17:43,878][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:17:44,379][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:17:44,874][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:17:45,372][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:17:45,874][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:17:46,370][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:17:46,868][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:17:47,364][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:17:47,864][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:17:48,360][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:17:48,858][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:17:49,356][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:17:49,855][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:17:50,353][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:17:50,848][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:17:51,349][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:17:51,849][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:17:52,353][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:17:52,855][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:17:53,357][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:17:53,858][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:17:54,359][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:17:54,863][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:17:55,364][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:17:55,866][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:17:56,371][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:17:56,872][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:17:57,372][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:17:57,876][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:17:58,375][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:17:58,873][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:17:59,370][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:17:59,866][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:18:00,365][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:18:00,862][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:18:01,358][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:18:01,855][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:18:02,356][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:18:02,858][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:18:03,358][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:18:03,856][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:18:04,355][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:18:04,854][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:18:05,358][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:18:05,860][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:18:06,364][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:18:06,865][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:18:07,365][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:18:07,867][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:18:08,369][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:18:08,871][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:18:09,377][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:18:09,881][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:18:10,393][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:18:10,908][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:18:11,404][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:18:11,905][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:18:12,406][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:18:12,910][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:18:13,432][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:18:13,933][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10464 tokens.
[2025-11-13 01:18:14,653][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 01:18:15,435][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:18:15,437][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:18:15,439][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:18:16,393][__main__][INFO] - Iteration 213 took 53s (32.90% Gen, 65.32% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 34m 39s. Estimated total time: 44h 43m 13s. Time estimates for 10 more iterations: 8m 56s, 100 more iterations: 1h 29m 26s, 500 more iterations: 7h 27m 12s.
[2025-11-13 01:18:16,396][__main__][INFO] - Starting iteration 213.
[2025-11-13 01:18:16,872][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:18:16,873][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:18:35,004][__main__][INFO] - Number of regex retries in iteration 213: 0
[2025-11-13 01:18:35,004][__main__][INFO] - agents played in iteration 213 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:18:35,841][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:18:35,866][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:18:35,891][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:18:35,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:18:35,913][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:18:35,914][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:18:36,579][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:18:37,038][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:18:37,544][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:18:38,041][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:18:38,538][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:18:39,036][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:18:39,546][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:18:40,049][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:18:40,544][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:18:41,043][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:18:41,541][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:18:42,041][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:18:42,539][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:18:43,040][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:18:43,549][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:18:44,051][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:18:44,552][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:18:45,055][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:18:45,554][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:18:46,061][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:18:46,564][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:18:47,066][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:18:47,572][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:18:48,072][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:18:48,573][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:18:49,074][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:18:49,574][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:18:50,080][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:18:50,580][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:18:51,082][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:18:51,586][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:18:52,086][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:18:52,585][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:18:53,091][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:18:53,591][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:18:54,105][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:18:54,605][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:18:55,110][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:18:55,610][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:18:56,112][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:18:56,611][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:18:57,111][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:18:57,611][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:18:58,112][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:18:58,615][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:18:59,131][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:18:59,631][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:19:00,133][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:19:00,649][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:19:01,151][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:19:01,656][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:19:02,159][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:19:02,662][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:19:03,165][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:19:03,666][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:19:04,172][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:19:04,670][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:19:05,170][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:19:05,673][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:19:06,172][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:19:06,672][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:19:07,176][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:19:07,677][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:19:08,180][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:19:08,681][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10576 tokens.
[2025-11-13 01:19:09,387][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.27%, Current % of VRAM taken: 60.53%, Block Peak % of device VRAM: 62.53%, ΔTime: 00:00:32
[2025-11-13 01:19:10,129][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:19:10,131][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:19:10,133][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:19:11,027][__main__][INFO] - Iteration 214 took 54s (33.48% Gen, 64.87% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 58m 16s. Estimated total time: 45h 7m 45s. Time estimates for 10 more iterations: 9m 1s, 100 more iterations: 1h 30m 15s, 500 more iterations: 7h 31m 17s.
[2025-11-13 01:19:11,029][__main__][INFO] - Starting iteration 214.
[2025-11-13 01:19:11,504][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:19:11,505][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:19:29,248][__main__][INFO] - Number of regex retries in iteration 214: 0
[2025-11-13 01:19:29,249][__main__][INFO] - agents played in iteration 214 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:19:30,102][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:19:30,129][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:19:30,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:19:30,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:19:30,179][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:19:30,180][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:19:30,832][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:19:31,289][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:19:31,796][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:19:32,293][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:19:32,795][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:19:33,299][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:19:33,799][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:19:34,298][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:19:34,803][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:19:35,304][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:19:35,820][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:19:36,323][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:19:36,823][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:19:37,329][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:19:37,833][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:19:38,343][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:19:38,844][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:19:39,348][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:19:39,850][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:19:40,352][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:19:40,859][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:19:41,360][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:19:41,860][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:19:42,360][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:19:42,856][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:19:43,360][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:19:43,860][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:19:44,360][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:19:44,860][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:19:45,362][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:19:45,865][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:19:46,371][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:19:46,869][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:19:47,373][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:19:47,871][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:19:48,368][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:19:48,865][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:19:49,365][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:19:49,862][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:19:50,366][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:19:50,864][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:19:51,364][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:19:51,861][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:19:52,360][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:19:52,862][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:19:53,361][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:19:53,858][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:19:54,359][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:19:54,856][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:19:55,356][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:19:55,855][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:19:56,353][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:19:56,852][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:19:57,349][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:19:57,850][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:19:58,351][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:19:58,850][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:19:59,361][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:19:59,856][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:20:00,352][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:20:00,851][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:20:01,350][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:20:01,852][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:20:02,351][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:20:02,850][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10406 tokens.
[2025-11-13 01:20:03,624][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 01:20:04,388][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:20:04,390][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:20:04,391][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:20:05,289][__main__][INFO] - Iteration 215 took 53s (32.99% Gen, 65.34% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 38m 50s. Estimated total time: 44h 49m 14s. Time estimates for 10 more iterations: 8m 57s, 100 more iterations: 1h 29m 38s, 500 more iterations: 7h 28m 12s.
[2025-11-13 01:20:05,291][__main__][INFO] - Starting iteration 215.
[2025-11-13 01:20:05,783][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:20:05,783][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:20:23,337][__main__][INFO] - Number of regex retries in iteration 215: 0
[2025-11-13 01:20:23,337][__main__][INFO] - agents played in iteration 215 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:20:24,129][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:20:24,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:20:24,182][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:20:24,205][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:20:24,205][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:20:24,207][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:20:24,838][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:20:25,293][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:20:25,805][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:20:26,306][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:20:26,806][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:20:27,311][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:20:27,817][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:20:28,320][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:20:28,821][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:20:29,325][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:20:29,839][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:20:30,345][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:20:30,858][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:20:31,359][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:20:31,862][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:20:32,363][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:20:32,867][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:20:33,370][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:20:33,871][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:20:34,371][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:20:34,878][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:20:35,378][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:20:35,882][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:20:36,385][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:20:36,888][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:20:37,388][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:20:37,893][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:20:38,397][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:20:38,899][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:20:39,398][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:20:39,899][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:20:40,400][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:20:40,900][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:20:41,399][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:20:41,897][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:20:42,394][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:20:42,893][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:20:43,392][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:20:43,894][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:20:44,397][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:20:44,896][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:20:45,395][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:20:45,893][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:20:46,395][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:20:46,901][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:20:47,402][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:20:47,899][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:20:48,396][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:20:48,894][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:20:49,394][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:20:49,891][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:20:50,391][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:20:50,889][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:20:51,385][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:20:51,885][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:20:52,385][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:20:52,883][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:20:53,388][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:20:53,891][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:20:54,392][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:20:54,896][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:20:55,394][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:20:55,896][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:20:56,396][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:20:56,897][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10540 tokens.
[2025-11-13 01:20:57,665][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.46%, ΔTime: 00:00:32
[2025-11-13 01:20:58,407][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:20:58,409][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:20:58,410][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:20:59,303][__main__][INFO] - Iteration 216 took 53s (32.80% Gen, 65.53% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 24m 44s. Estimated total time: 44h 36m 2s. Time estimates for 10 more iterations: 8m 55s, 100 more iterations: 1h 29m 12s, 500 more iterations: 7h 26m 0s.
[2025-11-13 01:20:59,305][__main__][INFO] - Starting iteration 216.
[2025-11-13 01:20:59,799][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:20:59,799][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:21:17,504][__main__][INFO] - Number of regex retries in iteration 216: 0
[2025-11-13 01:21:17,504][__main__][INFO] - agents played in iteration 216 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:21:18,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:21:18,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:21:18,345][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:21:18,367][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:21:18,368][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:21:18,369][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:21:19,097][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:21:19,559][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:21:20,078][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:21:20,584][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:21:21,088][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:21:21,586][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:21:22,087][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:21:22,591][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:21:23,094][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:21:23,594][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:21:24,096][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:21:24,598][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:21:25,104][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:21:25,604][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:21:26,106][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:21:26,609][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:21:27,110][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:21:27,610][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:21:28,110][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:21:28,611][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:21:29,111][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:21:29,612][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:21:30,111][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:21:30,613][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:21:31,110][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:21:31,619][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:21:32,120][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:21:32,619][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:21:33,118][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:21:33,618][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:21:34,119][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:21:34,624][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:21:35,128][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:21:35,629][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:21:36,132][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:21:36,630][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:21:37,138][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:21:37,641][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:21:38,140][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:21:38,645][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:21:39,149][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:21:39,671][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:21:40,174][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:21:40,676][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:21:41,173][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:21:41,671][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:21:42,182][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:21:42,683][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:21:43,176][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:21:43,692][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:21:44,194][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:21:44,693][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:21:45,193][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:21:45,698][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:21:46,202][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:21:46,698][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:21:47,200][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:21:47,705][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:21:48,206][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:21:48,709][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:21:49,209][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:21:49,711][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:21:50,212][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:21:50,708][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:21:51,209][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10455 tokens.
[2025-11-13 01:21:51,944][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 01:21:52,722][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:21:52,724][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:21:52,725][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:21:53,695][__main__][INFO] - Iteration 217 took 53s (32.85% Gen, 65.35% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 42m 37s. Estimated total time: 44h 54m 49s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 49s, 500 more iterations: 7h 29m 8s.
[2025-11-13 01:21:53,697][__main__][INFO] - Starting iteration 217.
[2025-11-13 01:21:54,208][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:21:54,209][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:22:12,721][__main__][INFO] - Number of regex retries in iteration 217: 0
[2025-11-13 01:22:12,721][__main__][INFO] - agents played in iteration 217 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:22:13,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:22:13,598][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:22:13,634][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:22:13,657][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:22:13,658][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:22:13,659][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:22:14,369][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:22:14,828][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:22:15,339][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:22:15,842][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:22:16,348][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:22:16,849][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:22:17,352][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:22:17,859][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:22:18,363][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:22:18,860][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:22:19,369][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:22:19,864][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:22:20,364][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:22:20,861][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:22:21,358][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:22:21,857][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:22:22,354][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:22:22,851][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:22:23,352][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:22:23,852][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:22:24,357][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:22:24,855][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:22:25,354][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:22:25,854][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:22:26,352][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:22:26,854][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:22:27,356][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:22:27,858][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:22:28,361][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:22:28,859][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:22:29,359][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:22:29,861][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:22:30,361][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:22:30,873][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:22:31,372][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:22:31,871][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:22:32,370][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:22:32,869][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:22:33,372][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:22:33,873][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:22:34,373][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:22:34,896][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:22:35,390][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:22:35,885][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:22:36,387][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:22:36,883][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:22:37,387][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:22:37,887][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:22:38,386][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:22:38,884][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:22:39,384][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:22:39,880][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:22:40,378][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:22:40,877][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:22:41,389][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:22:41,887][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:22:42,386][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:22:42,886][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:22:43,383][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:22:43,882][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:22:44,379][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:22:44,876][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:22:45,384][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:22:45,884][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:22:46,379][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10458 tokens.
[2025-11-13 01:22:47,115][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 01:22:47,895][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:22:47,897][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:22:47,899][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:22:49,038][__main__][INFO] - Iteration 218 took 54s (33.76% Gen, 64.16% Train). Generation: 18s, Training: 35s. Estimated remaining time: 42h 28m 21s. Estimated total time: 45h 41m 29s. Time estimates for 10 more iterations: 9m 8s, 100 more iterations: 1h 31m 22s, 500 more iterations: 7h 36m 54s.
[2025-11-13 01:22:49,040][__main__][INFO] - Starting iteration 218.
[2025-11-13 01:22:49,510][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:22:49,511][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:22:53,897][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:23:07,431][__main__][INFO] - Number of regex retries in iteration 218: 1
[2025-11-13 01:23:07,432][__main__][INFO] - agents played in iteration 218 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:23:08,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:23:08,309][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:23:08,331][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:23:08,353][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:23:08,354][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:23:08,354][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:23:09,057][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:23:09,517][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:23:10,026][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:23:10,531][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:23:11,032][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:23:11,532][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:23:12,032][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:23:12,528][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:23:13,026][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:23:13,528][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:23:14,027][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:23:14,524][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:23:15,022][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:23:15,522][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:23:16,020][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:23:16,521][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:23:17,021][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:23:17,522][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:23:18,028][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:23:18,528][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:23:19,034][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:23:19,533][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:23:20,033][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:23:20,530][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:23:21,026][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:23:21,522][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:23:22,022][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:23:22,520][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:23:23,019][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:23:23,518][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:23:24,018][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:23:24,517][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:23:25,019][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:23:25,517][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:23:26,040][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:23:26,539][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:23:27,036][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:23:27,536][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:23:28,041][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:23:28,539][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:23:29,039][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:23:29,537][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:23:30,035][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:23:30,541][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:23:31,043][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:23:31,540][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:23:32,043][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:23:32,537][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:23:33,034][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:23:33,535][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:23:34,034][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:23:34,538][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:23:35,038][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:23:35,542][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:23:36,041][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:23:36,542][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:23:37,047][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:23:37,544][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:23:38,040][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:23:38,540][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:23:39,039][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:23:39,540][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:23:40,036][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:23:40,538][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:23:41,034][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10368 tokens.
[2025-11-13 01:23:41,755][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.38%, ΔTime: 00:00:32
[2025-11-13 01:23:42,517][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:23:42,519][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:23:42,521][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:23:43,589][__main__][INFO] - Iteration 219 took 54s (33.14% Gen, 64.88% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 49m 56s. Estimated total time: 45h 3m 58s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 7s, 500 more iterations: 7h 30m 39s.
[2025-11-13 01:23:43,592][__main__][INFO] - Starting iteration 219.
[2025-11-13 01:23:44,077][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:23:44,077][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:24:00,512][__main__][INFO] - Number of regex retries in iteration 219: 0
[2025-11-13 01:24:00,513][__main__][INFO] - agents played in iteration 219 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:24:01,310][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:24:01,334][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:24:01,356][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:24:01,378][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:24:01,379][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:24:01,380][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:24:02,114][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:24:02,572][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:24:03,082][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:24:03,589][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:24:04,094][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:24:04,596][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:24:05,097][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:24:05,598][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:24:06,109][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:24:06,607][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:24:07,106][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:24:07,616][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:24:08,121][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:24:08,636][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:24:09,138][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:24:09,642][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:24:10,145][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:24:10,643][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:24:11,138][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:24:11,638][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:24:12,134][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:24:12,646][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:24:13,151][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:24:13,655][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:24:14,159][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:24:14,657][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:24:15,158][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:24:15,661][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:24:16,160][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:24:16,661][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:24:17,156][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:24:17,654][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:24:18,148][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:24:18,646][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:24:19,145][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:24:19,644][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:24:20,138][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:24:20,634][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:24:21,132][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:24:21,631][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:24:22,127][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:24:22,624][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:24:23,132][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:24:23,629][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:24:24,126][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:24:24,624][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:24:25,123][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:24:25,628][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:24:26,131][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:24:26,633][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:24:27,148][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:24:27,645][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:24:28,147][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:24:28,648][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:24:29,147][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:24:29,652][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:24:30,149][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:24:30,651][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:24:31,157][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:24:31,658][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:24:32,159][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:24:32,657][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:24:33,155][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:24:33,656][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:24:34,156][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10392 tokens.
[2025-11-13 01:24:34,864][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 01:24:35,635][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:24:35,637][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:24:35,639][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:24:36,553][__main__][INFO] - Iteration 220 took 52s (31.32% Gen, 66.94% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 28m 56s. Estimated total time: 43h 43m 51s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 27s, 500 more iterations: 7h 17m 18s.
[2025-11-13 01:24:36,555][__main__][INFO] - Starting iteration 220.
[2025-11-13 01:24:37,052][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 01:24:37,053][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:24:40,710][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:24:53,961][__main__][INFO] - Number of regex retries in iteration 220: 1
[2025-11-13 01:24:53,962][__main__][INFO] - agents played in iteration 220 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:24:54,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:24:54,781][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:24:54,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:24:54,829][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:24:54,829][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:24:54,831][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:24:55,509][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:24:56,104][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:24:56,609][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:24:57,110][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:24:57,622][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:24:58,120][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:24:58,632][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:24:59,137][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:24:59,641][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:25:00,145][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:25:00,645][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:25:01,152][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:25:01,656][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:25:02,156][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:25:02,663][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:25:03,160][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:25:03,658][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:25:04,156][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:25:04,658][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:25:05,163][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:25:05,663][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:25:06,161][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:25:06,658][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:25:07,155][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:25:07,654][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:25:08,152][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:25:08,648][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:25:09,157][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:25:09,653][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:25:10,148][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:25:10,644][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:25:11,141][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:25:11,647][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:25:12,148][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:25:12,651][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:25:13,152][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:25:13,650][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:25:14,149][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:25:14,647][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:25:15,143][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:25:15,657][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:25:16,152][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:25:16,652][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:25:17,151][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:25:17,646][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:25:18,154][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:25:18,657][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:25:19,156][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:25:19,661][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:25:20,168][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:25:20,668][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:25:21,170][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:25:21,674][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:25:22,177][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:25:22,677][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:25:23,179][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:25:23,681][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:25:24,179][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:25:24,678][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:25:25,176][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:25:25,672][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:25:26,169][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:25:26,665][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:25:27,162][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:25:27,657][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10389 tokens.
[2025-11-13 01:25:28,362][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 01:25:29,149][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:25:29,150][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:25:29,152][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:25:31,259][__main__][INFO] - Iteration 221 took 54s (31.19% Gen, 64.92% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 54m 32s. Estimated total time: 45h 10m 22s. Time estimates for 10 more iterations: 9m 2s, 100 more iterations: 1h 30m 20s, 500 more iterations: 7h 31m 43s.
[2025-11-13 01:25:31,261][__main__][INFO] - Starting iteration 221.
[2025-11-13 01:25:31,763][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:25:31,764][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:25:49,858][__main__][INFO] - Number of regex retries in iteration 221: 0
[2025-11-13 01:25:49,859][__main__][INFO] - agents played in iteration 221 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:25:50,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:25:50,668][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:25:50,705][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:25:50,729][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:25:50,730][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:25:50,731][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:25:51,399][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:25:51,858][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:25:52,362][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:25:52,859][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:25:53,357][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:25:53,857][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:25:54,353][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:25:54,872][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:25:55,370][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:25:55,872][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:25:56,370][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:25:56,873][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:25:57,375][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:25:57,878][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:25:58,377][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:25:58,879][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:25:59,376][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:25:59,874][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:26:00,370][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:26:00,868][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:26:01,367][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:26:01,864][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:26:02,361][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:26:02,873][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:26:03,373][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:26:03,872][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:26:04,367][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:26:04,863][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:26:05,363][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:26:05,860][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:26:06,355][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:26:06,855][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:26:07,354][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:26:07,854][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:26:08,352][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:26:08,850][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:26:09,362][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:26:09,860][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:26:10,357][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:26:10,856][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:26:11,354][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:26:11,864][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:26:12,362][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:26:12,861][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:26:13,362][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:26:13,859][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:26:14,362][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:26:14,864][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:26:15,363][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:26:15,860][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:26:16,359][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:26:16,858][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:26:17,375][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:26:17,874][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:26:18,371][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:26:18,873][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:26:19,367][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:26:19,866][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:26:20,364][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:26:20,860][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:26:21,359][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:26:21,856][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:26:22,359][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:26:22,858][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:26:23,362][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10364 tokens.
[2025-11-13 01:26:24,120][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 01:26:24,919][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:26:24,921][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:26:24,922][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:26:25,882][__main__][INFO] - Iteration 222 took 54s (33.43% Gen, 64.79% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 49m 15s. Estimated total time: 45h 5m 59s. Time estimates for 10 more iterations: 9m 1s, 100 more iterations: 1h 30m 11s, 500 more iterations: 7h 30m 59s.
[2025-11-13 01:26:25,885][__main__][INFO] - Starting iteration 222.
[2025-11-13 01:26:26,364][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:26:26,365][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:26:30,814][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:26:44,209][__main__][INFO] - Number of regex retries in iteration 222: 1
[2025-11-13 01:26:44,209][__main__][INFO] - agents played in iteration 222 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:26:44,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:26:45,016][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:26:45,042][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:26:45,064][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:26:45,065][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:26:45,066][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:26:45,731][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:26:46,186][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:26:46,686][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:26:47,190][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:26:47,692][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:26:48,195][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:26:48,696][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:26:49,194][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:26:49,692][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:26:50,186][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:26:50,683][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:26:51,186][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:26:51,680][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:26:52,181][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:26:52,675][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:26:53,171][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:26:53,673][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:26:54,168][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:26:54,666][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:26:55,162][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:26:55,658][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:26:56,153][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:26:56,646][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:26:57,140][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:26:57,643][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:26:58,139][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:26:58,637][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:26:59,135][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:26:59,636][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:27:00,137][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:27:00,633][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:27:01,130][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:27:01,627][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:27:02,129][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:27:02,627][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:27:03,122][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:27:03,620][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:27:04,121][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:27:04,620][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:27:05,118][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:27:05,620][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:27:06,117][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:27:06,614][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:27:07,113][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:27:07,615][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:27:08,114][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:27:08,613][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:27:09,115][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:27:09,616][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:27:10,113][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:27:10,608][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:27:11,104][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:27:11,601][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:27:12,100][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:27:12,596][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:27:13,092][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:27:13,592][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:27:14,092][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:27:14,591][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:27:15,093][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:27:15,591][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:27:16,097][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:27:16,597][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:27:17,098][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:27:17,597][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10443 tokens.
[2025-11-13 01:27:18,324][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 01:27:19,080][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:27:19,083][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:27:19,085][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:27:19,969][__main__][INFO] - Iteration 223 took 53s (33.29% Gen, 65.06% Train). Generation: 17s, Training: 34s. Estimated remaining time: 41h 22m 37s. Estimated total time: 44h 40m 15s. Time estimates for 10 more iterations: 8m 56s, 100 more iterations: 1h 29m 20s, 500 more iterations: 7h 26m 42s.
[2025-11-13 01:27:19,971][__main__][INFO] - Starting iteration 223.
[2025-11-13 01:27:20,461][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:27:20,462][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:27:33,704][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 10 books, 20 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:27:35,109][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 10 books, 20 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 2/3
[2025-11-13 01:27:38,223][__main__][INFO] - Number of regex retries in iteration 223: 2
[2025-11-13 01:27:38,224][__main__][INFO] - agents played in iteration 223 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:27:39,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:27:39,082][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:27:39,109][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:27:39,132][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:27:39,133][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:27:39,134][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:27:39,839][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:27:40,291][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:27:40,799][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:27:41,302][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:27:41,806][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:27:42,302][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:27:42,797][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:27:43,294][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:27:43,796][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:27:44,293][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:27:44,792][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:27:45,294][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:27:45,794][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:27:46,294][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:27:46,792][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:27:47,291][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:27:47,792][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:27:48,291][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:27:48,791][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:27:49,287][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:27:49,785][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:27:50,294][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:27:50,795][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:27:51,293][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:27:51,794][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:27:52,289][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:27:52,787][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:27:53,292][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:27:53,789][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:27:54,286][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:27:54,782][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:27:55,278][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:27:55,787][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:27:56,287][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:27:56,791][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:27:57,293][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:27:57,794][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:27:58,298][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:27:58,800][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:27:59,299][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:27:59,802][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:28:00,301][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:28:00,803][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:28:01,304][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:28:01,808][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:28:02,310][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:28:02,813][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:28:03,311][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:28:03,815][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:28:04,311][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:28:04,809][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:28:05,307][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:28:05,805][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:28:06,306][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:28:06,805][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:28:07,304][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:28:07,806][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:28:08,304][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:28:08,805][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:28:09,305][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:28:09,808][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:28:10,309][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:28:10,808][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:28:11,309][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:28:11,814][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10310 tokens.
[2025-11-13 01:28:12,575][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 01:28:13,356][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:28:13,358][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:28:13,360][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:28:14,321][__main__][INFO] - Iteration 224 took 53s (32.98% Gen, 65.23% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 34m 31s. Estimated total time: 44h 53m 3s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 46s, 500 more iterations: 7h 28m 50s.
[2025-11-13 01:28:14,323][__main__][INFO] - Starting iteration 224.
[2025-11-13 01:28:14,801][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:28:14,802][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:28:19,342][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:28:32,973][__main__][INFO] - Number of regex retries in iteration 224: 1
[2025-11-13 01:28:32,973][__main__][INFO] - agents played in iteration 224 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:28:33,846][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:28:33,871][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:28:33,899][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:28:33,921][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:28:33,922][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:28:33,924][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:28:34,600][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:28:35,055][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:28:35,560][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:28:36,060][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:28:36,577][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:28:37,075][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:28:37,575][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:28:38,074][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:28:38,575][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:28:39,080][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:28:39,574][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:28:40,071][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:28:40,585][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:28:41,080][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:28:41,575][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:28:42,073][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:28:42,571][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:28:43,064][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:28:43,559][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:28:44,055][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:28:44,567][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:28:45,061][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:28:45,557][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:28:46,058][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:28:46,554][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:28:47,059][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:28:47,554][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:28:48,053][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:28:48,576][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:28:49,077][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:28:49,578][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:28:50,081][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:28:50,582][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:28:51,088][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:28:51,583][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:28:52,082][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:28:52,591][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:28:53,089][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:28:53,587][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:28:54,086][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:28:54,586][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:28:55,088][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:28:55,588][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:28:56,083][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:28:56,580][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:28:57,075][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:28:57,571][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:28:58,070][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:28:58,567][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:28:59,090][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:28:59,587][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:29:00,087][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:29:00,586][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:29:01,085][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:29:01,590][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:29:02,088][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:29:02,588][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:29:03,086][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:29:03,582][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:29:04,079][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:29:04,579][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:29:05,080][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:29:05,577][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:29:06,075][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:29:06,576][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10343 tokens.
[2025-11-13 01:29:07,340][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 01:29:08,120][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:29:08,122][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:29:08,123][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:29:09,049][__main__][INFO] - Iteration 225 took 54s (33.50% Gen, 64.80% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 52m 57s. Estimated total time: 45h 12m 24s. Time estimates for 10 more iterations: 9m 2s, 100 more iterations: 1h 30m 24s, 500 more iterations: 7h 32m 4s.
[2025-11-13 01:29:09,053][__main__][INFO] - Starting iteration 225.
[2025-11-13 01:29:09,528][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:29:09,529][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:29:13,697][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:29:27,367][__main__][INFO] - Number of regex retries in iteration 225: 1
[2025-11-13 01:29:27,367][__main__][INFO] - agents played in iteration 225 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:29:28,215][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:29:28,239][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:29:28,263][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:29:28,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:29:28,286][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:29:28,288][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:29:28,951][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:29:29,413][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:29:29,913][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:29:30,413][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:29:30,915][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:29:31,411][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:29:31,909][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:29:32,412][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:29:32,910][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:29:33,408][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:29:33,908][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:29:34,413][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:29:34,908][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:29:35,410][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:29:35,908][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:29:36,403][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:29:36,908][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:29:37,404][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:29:37,900][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:29:38,400][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:29:38,896][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:29:39,391][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:29:39,889][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:29:40,390][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:29:40,910][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:29:41,409][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:29:41,909][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:29:42,408][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:29:42,907][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:29:43,415][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:29:43,915][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:29:44,416][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:29:44,918][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:29:45,419][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:29:45,918][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:29:46,415][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:29:46,914][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:29:47,418][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:29:47,916][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:29:48,417][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:29:48,913][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:29:49,412][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:29:49,916][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:29:50,417][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:29:50,918][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:29:51,418][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:29:51,918][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:29:52,417][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:29:52,917][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:29:53,414][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:29:53,924][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:29:54,423][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:29:54,925][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:29:55,425][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:29:55,925][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:29:56,427][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:29:56,925][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:29:57,426][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:29:57,929][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:29:58,431][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:29:58,933][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:29:59,434][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:29:59,933][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:30:00,437][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:30:00,938][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10275 tokens.
[2025-11-13 01:30:01,655][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 01:30:02,463][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:30:02,465][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:30:02,466][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:30:03,339][__main__][INFO] - Iteration 226 took 53s (33.15% Gen, 65.23% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 30m 13s. Estimated total time: 44h 50m 35s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 41s, 500 more iterations: 7h 28m 25s.
[2025-11-13 01:30:03,341][__main__][INFO] - Starting iteration 226.
[2025-11-13 01:30:03,812][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:30:03,813][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:30:21,645][__main__][INFO] - Number of regex retries in iteration 226: 0
[2025-11-13 01:30:21,646][__main__][INFO] - agents played in iteration 226 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:30:22,430][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:30:22,460][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:30:22,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:30:22,510][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:30:22,511][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:30:22,512][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:30:23,152][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:30:23,604][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:30:24,111][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:30:24,608][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:30:25,107][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:30:25,607][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:30:26,103][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:30:26,599][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:30:27,101][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:30:27,597][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:30:28,098][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:30:28,595][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:30:29,105][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:30:29,601][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:30:30,098][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:30:30,611][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:30:31,106][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:30:31,605][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:30:32,135][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:30:32,637][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:30:33,142][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:30:33,646][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:30:34,148][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:30:34,651][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:30:35,150][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:30:35,649][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:30:36,147][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:30:36,645][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:30:37,144][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:30:37,642][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:30:38,140][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:30:38,641][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:30:39,137][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:30:39,636][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:30:40,133][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:30:40,629][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:30:41,133][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:30:41,628][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:30:42,127][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:30:42,631][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:30:43,127][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:30:43,630][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:30:44,129][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:30:44,629][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:30:45,136][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:30:45,637][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:30:46,135][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:30:46,640][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:30:47,143][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:30:47,643][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:30:48,153][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:30:48,654][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:30:49,157][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:30:49,667][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:30:50,165][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:30:50,665][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:30:51,165][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:30:51,665][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:30:52,169][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:30:52,671][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:30:53,173][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:30:53,677][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:30:54,180][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:30:54,682][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:30:55,187][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10429 tokens.
[2025-11-13 01:30:56,027][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 01:30:56,758][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:30:56,760][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:30:56,762][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:30:57,639][__main__][INFO] - Iteration 227 took 53s (33.13% Gen, 65.24% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 30m 6s. Estimated total time: 44h 51m 22s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 42s, 500 more iterations: 7h 28m 33s.
[2025-11-13 01:30:57,641][__main__][INFO] - Starting iteration 227.
[2025-11-13 01:30:58,109][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:30:58,110][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:31:14,766][__main__][INFO] - Number of regex retries in iteration 227: 0
[2025-11-13 01:31:14,767][__main__][INFO] - agents played in iteration 227 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:31:15,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:31:15,619][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:31:15,644][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:31:15,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:31:15,668][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:31:15,669][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:31:16,322][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:31:16,773][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:31:17,280][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:31:17,781][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:31:18,278][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:31:18,773][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:31:19,268][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:31:19,768][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:31:20,262][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:31:20,757][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:31:21,259][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:31:21,755][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:31:22,255][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:31:22,751][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:31:23,246][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:31:23,746][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:31:24,244][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:31:24,741][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:31:25,240][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:31:25,740][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:31:26,238][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:31:26,739][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:31:27,240][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:31:27,740][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:31:28,241][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:31:28,738][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:31:29,235][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:31:29,730][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:31:30,238][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:31:30,736][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:31:31,234][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:31:31,733][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:31:32,230][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:31:32,729][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:31:33,237][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:31:33,738][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:31:34,240][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:31:34,742][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:31:35,241][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:31:35,739][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:31:36,235][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:31:36,738][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:31:37,237][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:31:37,735][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:31:38,242][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:31:38,738][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:31:39,239][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:31:39,739][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:31:40,236][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:31:40,738][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:31:41,237][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:31:41,735][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:31:42,242][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:31:42,741][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:31:43,242][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:31:43,740][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:31:44,243][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:31:44,746][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:31:45,245][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:31:45,746][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:31:46,249][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:31:46,745][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:31:47,244][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:31:47,743][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:31:48,241][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10432 tokens.
[2025-11-13 01:31:48,985][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:32
[2025-11-13 01:31:49,725][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:31:49,727][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:31:49,728][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:31:50,603][__main__][INFO] - Iteration 228 took 52s (31.73% Gen, 66.60% Train). Generation: 16s, Training: 34s. Estimated remaining time: 40h 22m 34s. Estimated total time: 43h 44m 43s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 29s, 500 more iterations: 7h 17m 27s.
[2025-11-13 01:31:50,605][__main__][INFO] - Starting iteration 228.
[2025-11-13 01:31:51,083][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:31:51,084][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:31:56,211][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:32:07,041][__main__][INFO] - Number of regex retries in iteration 228: 1
[2025-11-13 01:32:07,041][__main__][INFO] - agents played in iteration 228 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:32:07,852][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:32:07,880][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:32:07,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:32:07,929][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:32:07,929][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:32:07,930][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:32:08,579][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:32:09,029][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:32:09,534][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:32:10,049][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:32:10,544][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:32:11,044][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:32:11,540][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:32:12,036][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:32:12,539][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:32:13,033][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:32:13,528][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:32:14,043][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:32:14,537][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:32:15,032][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:32:15,530][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:32:16,024][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:32:16,532][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:32:17,025][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:32:17,517][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:32:18,029][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:32:18,526][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:32:19,024][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:32:19,522][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:32:20,019][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:32:20,525][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:32:21,021][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:32:21,522][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:32:22,036][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:32:22,534][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:32:23,032][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:32:23,530][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:32:24,029][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:32:24,526][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:32:25,026][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:32:25,523][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:32:26,026][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:32:26,528][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:32:27,036][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:32:27,537][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:32:28,038][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:32:28,538][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:32:29,037][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:32:29,536][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:32:30,036][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:32:30,535][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:32:31,051][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:32:31,553][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:32:32,052][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:32:32,552][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:32:33,053][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:32:33,551][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:32:34,048][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:32:34,547][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:32:35,056][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:32:35,552][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:32:36,051][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:32:36,553][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:32:37,055][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:32:37,559][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:32:38,067][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:32:38,567][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:32:39,071][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:32:39,575][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:32:40,080][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:32:40,583][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10274 tokens.
[2025-11-13 01:32:41,370][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.35%, ΔTime: 00:00:32
[2025-11-13 01:32:42,101][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:32:42,103][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:32:42,104][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:32:42,978][__main__][INFO] - Iteration 229 took 51s (30.75% Gen, 67.57% Train). Generation: 15s, Training: 35s. Estimated remaining time: 39h 51m 42s. Estimated total time: 43h 14m 43s. Time estimates for 10 more iterations: 8m 38s, 100 more iterations: 1h 26m 29s, 500 more iterations: 7h 12m 27s.
[2025-11-13 01:32:42,980][__main__][INFO] - Starting iteration 229.
[2025-11-13 01:32:43,465][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:32:43,466][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:32:54,326][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:33:01,067][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 10 books, 1 z ball did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:33:01,937][__main__][INFO] - Number of regex retries in iteration 229: 2
[2025-11-13 01:33:01,937][__main__][INFO] - agents played in iteration 229 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:33:02,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:33:02,777][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:33:02,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:33:02,825][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:33:02,825][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:33:02,826][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:33:03,461][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:33:03,937][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:33:04,438][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:33:04,934][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:33:05,438][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:33:05,935][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:33:06,443][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:33:06,941][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:33:07,437][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:33:07,947][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:33:08,441][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:33:08,942][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:33:09,442][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:33:09,937][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:33:10,446][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:33:10,948][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:33:11,446][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:33:11,943][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:33:12,438][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:33:12,938][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:33:13,432][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:33:13,929][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:33:14,455][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:33:14,953][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:33:15,454][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:33:15,955][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:33:16,456][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:33:16,956][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:33:17,455][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:33:17,956][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:33:18,464][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:33:18,963][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:33:19,467][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:33:19,965][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:33:20,467][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:33:20,964][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:33:21,466][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:33:21,967][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:33:22,471][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:33:22,969][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:33:23,467][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:33:23,965][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:33:24,463][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:33:24,967][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:33:25,467][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:33:25,967][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:33:26,470][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:33:26,970][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:33:27,470][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:33:27,967][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:33:28,463][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:33:28,965][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:33:29,463][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:33:29,963][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:33:30,463][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:33:30,961][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:33:31,465][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:33:31,965][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:33:32,465][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:33:32,968][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:33:33,465][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:33:33,967][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:33:34,472][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:33:34,973][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:33:35,479][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10369 tokens.
[2025-11-13 01:33:36,200][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:32
[2025-11-13 01:33:36,968][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:33:36,970][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:33:36,972][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:33:37,921][__main__][INFO] - Iteration 230 took 54s (33.92% Gen, 64.34% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 58m 53s. Estimated total time: 45h 22m 49s. Time estimates for 10 more iterations: 9m 4s, 100 more iterations: 1h 30m 45s, 500 more iterations: 7h 33m 48s.
[2025-11-13 01:33:37,923][__main__][INFO] - Starting iteration 230.
[2025-11-13 01:33:38,408][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 01:33:38,409][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:33:56,084][__main__][INFO] - Number of regex retries in iteration 230: 0
[2025-11-13 01:33:56,084][__main__][INFO] - agents played in iteration 230 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:33:56,903][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:33:56,930][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:33:56,956][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:33:56,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:33:56,979][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:33:56,980][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:33:57,616][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:33:58,072][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:33:58,576][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:33:59,075][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:33:59,570][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:34:00,069][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:34:00,572][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:34:01,072][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:34:01,571][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:34:02,064][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:34:02,559][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:34:03,062][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:34:03,556][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:34:04,050][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:34:04,545][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:34:05,039][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:34:05,544][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:34:06,040][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:34:06,536][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:34:07,035][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:34:07,537][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:34:08,033][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:34:08,536][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:34:09,036][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:34:09,549][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:34:10,053][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:34:10,551][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:34:11,057][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:34:11,557][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:34:12,069][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:34:12,569][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:34:13,072][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:34:13,572][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:34:14,075][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:34:14,575][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:34:15,074][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:34:15,576][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:34:16,078][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:34:16,579][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:34:17,080][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:34:17,582][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:34:18,081][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:34:18,584][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:34:19,084][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:34:19,589][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:34:20,094][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:34:20,597][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:34:21,096][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:34:21,597][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:34:22,094][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:34:22,593][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:34:23,095][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:34:23,596][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:34:24,096][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:34:24,595][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:34:25,095][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:34:25,596][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:34:26,096][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:34:26,604][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:34:27,105][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:34:27,609][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:34:28,113][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:34:28,615][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:34:29,118][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:34:29,617][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10371 tokens.
[2025-11-13 01:34:30,330][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 01:34:31,102][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:34:31,104][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:34:31,107][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:34:32,932][__main__][INFO] - Iteration 231 took 54s (32.42% Gen, 64.23% Train). Generation: 17s, Training: 35s. Estimated remaining time: 42h 1m 20s. Estimated total time: 45h 26m 12s. Time estimates for 10 more iterations: 9m 5s, 100 more iterations: 1h 30m 52s, 500 more iterations: 7h 34m 22s.
[2025-11-13 01:34:32,934][__main__][INFO] - Starting iteration 231.
[2025-11-13 01:34:33,417][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:34:33,418][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:34:37,809][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:34:51,079][__main__][INFO] - Number of regex retries in iteration 231: 1
[2025-11-13 01:34:51,080][__main__][INFO] - agents played in iteration 231 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:34:51,880][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:34:51,908][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:34:51,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:34:51,956][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:34:51,956][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:34:51,957][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:34:52,618][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:34:53,076][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:34:53,576][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:34:54,074][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:34:54,575][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:34:55,071][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:34:55,573][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:34:56,070][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:34:56,583][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:34:57,087][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:34:57,603][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:34:58,101][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:34:58,602][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:34:59,102][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:34:59,603][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:35:00,106][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:35:00,611][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:35:01,113][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:35:01,612][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:35:02,116][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:35:02,618][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:35:03,124][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:35:03,622][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:35:04,123][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:35:04,625][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:35:05,126][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:35:05,625][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:35:06,128][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:35:06,626][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:35:07,128][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:35:07,626][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:35:08,129][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:35:08,633][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:35:09,130][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:35:09,628][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:35:10,129][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:35:10,627][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:35:11,126][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:35:11,624][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:35:12,125][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:35:12,623][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:35:13,124][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:35:13,625][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:35:14,126][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:35:14,625][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:35:15,128][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:35:15,630][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:35:16,130][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:35:16,629][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:35:17,128][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:35:17,627][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:35:18,126][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:35:18,628][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:35:19,127][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:35:19,626][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:35:20,128][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:35:20,625][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:35:21,124][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:35:21,623][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:35:22,121][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:35:22,618][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:35:23,117][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:35:23,618][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:35:24,121][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:35:24,617][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10369 tokens.
[2025-11-13 01:35:25,297][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 01:35:26,044][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:35:26,047][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:35:26,049][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:35:26,970][__main__][INFO] - Iteration 232 took 53s (32.98% Gen, 65.30% Train). Generation: 17s, Training: 34s. Estimated remaining time: 41h 11m 52s. Estimated total time: 44h 37m 38s. Time estimates for 10 more iterations: 8m 55s, 100 more iterations: 1h 29m 15s, 500 more iterations: 7h 26m 16s.
[2025-11-13 01:35:26,972][__main__][INFO] - Starting iteration 232.
[2025-11-13 01:35:27,472][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:35:27,472][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:35:40,090][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:35:45,768][__main__][INFO] - Number of regex retries in iteration 232: 1
[2025-11-13 01:35:45,768][__main__][INFO] - agents played in iteration 232 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:35:46,587][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:35:46,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:35:46,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:35:46,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:35:46,664][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:35:46,665][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:35:47,300][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:35:47,774][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:35:48,274][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:35:48,776][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:35:49,274][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:35:49,769][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:35:50,270][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:35:50,766][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:35:51,264][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:35:51,786][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:35:52,282][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:35:52,778][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:35:53,280][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:35:53,782][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:35:54,290][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:35:54,790][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:35:55,299][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:35:55,804][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:35:56,301][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:35:56,807][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:35:57,311][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:35:57,811][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:35:58,315][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:35:58,816][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:35:59,317][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:35:59,818][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:36:00,318][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:36:00,815][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:36:01,317][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:36:01,815][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:36:02,318][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:36:02,818][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:36:03,318][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:36:03,818][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:36:04,319][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:36:04,823][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:36:05,319][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:36:05,819][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:36:06,320][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:36:06,820][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:36:07,318][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:36:07,815][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:36:08,315][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:36:08,819][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:36:09,318][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:36:09,820][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:36:10,320][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:36:10,825][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:36:11,324][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:36:11,822][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:36:12,323][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:36:12,829][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:36:13,328][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:36:13,825][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:36:14,330][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:36:14,833][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:36:15,333][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:36:15,832][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:36:16,329][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:36:16,824][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:36:17,321][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:36:17,817][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:36:18,315][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:36:18,810][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:36:19,307][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10296 tokens.
[2025-11-13 01:36:19,994][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 01:36:20,785][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:36:20,787][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:36:20,789][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:36:21,711][__main__][INFO] - Iteration 233 took 54s (33.73% Gen, 64.57% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 45m 18s. Estimated total time: 45h 11m 58s. Time estimates for 10 more iterations: 9m 2s, 100 more iterations: 1h 30m 23s, 500 more iterations: 7h 31m 59s.
[2025-11-13 01:36:21,713][__main__][INFO] - Starting iteration 233.
[2025-11-13 01:36:22,225][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:36:22,225][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:36:39,972][__main__][INFO] - Number of regex retries in iteration 233: 0
[2025-11-13 01:36:39,973][__main__][INFO] - agents played in iteration 233 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:36:40,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:36:40,835][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:36:40,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:36:40,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:36:40,885][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:36:40,886][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:36:41,550][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:36:42,008][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:36:42,513][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:36:43,011][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:36:43,510][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:36:44,006][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:36:44,504][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:36:45,008][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:36:45,506][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:36:46,006][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:36:46,507][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:36:47,006][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:36:47,505][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:36:48,003][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:36:48,504][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:36:49,013][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:36:49,514][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:36:50,017][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:36:50,522][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:36:51,023][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:36:51,522][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:36:52,022][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:36:52,528][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:36:53,029][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:36:53,528][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:36:54,026][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:36:54,523][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:36:55,020][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:36:55,522][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:36:56,020][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:36:56,521][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:36:57,015][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:36:57,514][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:36:58,015][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:36:58,516][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:36:59,017][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:36:59,519][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:37:00,022][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:37:00,522][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:37:01,021][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:37:01,522][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:37:02,024][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:37:02,525][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:37:03,026][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:37:03,529][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:37:04,031][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:37:04,538][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:37:05,072][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:37:05,577][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:37:06,084][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:37:06,582][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:37:07,085][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:37:07,583][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:37:08,082][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:37:08,584][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:37:09,082][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:37:09,580][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:37:10,089][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:37:10,588][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:37:11,097][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:37:11,597][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:37:12,094][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:37:12,592][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:37:13,092][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:37:13,603][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10330 tokens.
[2025-11-13 01:37:14,284][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 01:37:15,073][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:37:15,074][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:37:15,076][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:37:16,084][__main__][INFO] - Iteration 234 took 53s (32.95% Gen, 65.17% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 25m 25s. Estimated total time: 44h 52m 59s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 45s, 500 more iterations: 7h 28m 49s.
[2025-11-13 01:37:16,086][__main__][INFO] - Starting iteration 234.
[2025-11-13 01:37:16,560][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:37:16,561][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:37:25,501][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:37:34,220][__main__][INFO] - Number of regex retries in iteration 234: 1
[2025-11-13 01:37:34,220][__main__][INFO] - agents played in iteration 234 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:37:35,055][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:37:35,083][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:37:35,110][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:37:35,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:37:35,133][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:37:35,134][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:37:35,806][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:37:36,266][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:37:36,773][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:37:37,283][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:37:37,783][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:37:38,287][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:37:38,798][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:37:39,300][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:37:39,812][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:37:40,318][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:37:40,822][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:37:41,325][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:37:41,826][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:37:42,328][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:37:42,824][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:37:43,323][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:37:43,829][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:37:44,327][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:37:44,828][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:37:45,330][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:37:45,830][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:37:46,335][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:37:46,834][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:37:47,336][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:37:47,836][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:37:48,335][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:37:48,835][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:37:49,335][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:37:49,835][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:37:50,338][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:37:50,836][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:37:51,337][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:37:51,835][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:37:52,336][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:37:52,834][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:37:53,337][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:37:53,836][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:37:54,335][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:37:54,839][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:37:55,342][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:37:55,842][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:37:56,344][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:37:56,848][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:37:57,352][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:37:57,854][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:37:58,355][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:37:58,857][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:37:59,362][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:37:59,861][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:38:00,362][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:38:00,861][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:38:01,360][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:38:01,858][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:38:02,357][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:38:02,855][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:38:03,355][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:38:03,852][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:38:04,352][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:38:04,849][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:38:05,348][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:38:05,845][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:38:06,343][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:38:06,841][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:38:07,341][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:38:07,838][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10449 tokens.
[2025-11-13 01:38:08,529][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:32
[2025-11-13 01:38:09,277][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:38:09,278][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:38:09,280][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:38:10,261][__main__][INFO] - Iteration 235 took 53s (32.88% Gen, 65.28% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 16m 37s. Estimated total time: 44h 45m 5s. Time estimates for 10 more iterations: 8m 57s, 100 more iterations: 1h 29m 30s, 500 more iterations: 7h 27m 30s.
[2025-11-13 01:38:10,263][__main__][INFO] - Starting iteration 235.
[2025-11-13 01:38:10,730][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:38:10,731][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:38:28,632][__main__][INFO] - Number of regex retries in iteration 235: 0
[2025-11-13 01:38:28,632][__main__][INFO] - agents played in iteration 235 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:38:29,455][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:38:29,484][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:38:29,511][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:38:29,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:38:29,540][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:38:29,541][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:38:30,206][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:38:30,664][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:38:31,168][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:38:31,667][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:38:32,167][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:38:32,671][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:38:33,171][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:38:33,672][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:38:34,172][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:38:34,678][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:38:35,184][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:38:35,690][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:38:36,195][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:38:36,694][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:38:37,194][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:38:37,709][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:38:38,211][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:38:38,709][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:38:39,206][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:38:39,705][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:38:40,226][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:38:40,724][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:38:41,223][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:38:41,723][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:38:42,220][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:38:42,721][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:38:43,218][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:38:43,716][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:38:44,231][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:38:44,729][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:38:45,227][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:38:45,725][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:38:46,223][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:38:46,734][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:38:47,235][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:38:47,740][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:38:48,238][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:38:48,740][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:38:49,241][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:38:49,745][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:38:50,246][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:38:50,750][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:38:51,250][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:38:51,751][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:38:52,260][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:38:52,794][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:38:53,307][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:38:53,807][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:38:54,311][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:38:54,813][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:38:55,320][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:38:55,824][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:38:56,322][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:38:56,821][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:38:57,335][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:38:57,834][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:38:58,332][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:38:58,834][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:38:59,335][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:38:59,845][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:39:00,345][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:39:00,842][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:39:01,340][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:39:01,838][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:39:02,340][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10301 tokens.
[2025-11-13 01:39:03,020][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 01:39:03,957][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:39:03,958][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:39:03,975][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:39:05,123][__main__][INFO] - Iteration 236 took 54s (32.91% Gen, 64.98% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 50m 18s. Estimated total time: 45h 19m 41s. Time estimates for 10 more iterations: 9m 3s, 100 more iterations: 1h 30m 39s, 500 more iterations: 7h 33m 16s.
[2025-11-13 01:39:05,126][__main__][INFO] - Starting iteration 236.
[2025-11-13 01:39:05,592][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:39:05,592][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:39:22,852][__main__][INFO] - Number of regex retries in iteration 236: 0
[2025-11-13 01:39:22,852][__main__][INFO] - agents played in iteration 236 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:39:23,701][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:39:23,727][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:39:23,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:39:23,774][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:39:23,774][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:39:23,776][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:39:24,453][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:39:24,911][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:39:25,419][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:39:25,920][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:39:26,422][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:39:26,921][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:39:27,436][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:39:27,936][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:39:28,438][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:39:28,941][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:39:29,440][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:39:29,944][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:39:30,443][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:39:30,944][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:39:31,448][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:39:31,945][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:39:32,444][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:39:32,940][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:39:33,438][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:39:33,950][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:39:34,450][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:39:34,947][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:39:35,455][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:39:35,953][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:39:36,454][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:39:36,952][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:39:37,449][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:39:37,952][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:39:38,451][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:39:38,957][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:39:39,457][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:39:39,957][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:39:40,457][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:39:40,959][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:39:41,460][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:39:41,963][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:39:42,463][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:39:42,959][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:39:43,463][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:39:43,962][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:39:44,464][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:39:44,965][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:39:45,465][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:39:45,972][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:39:46,474][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:39:46,976][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:39:47,474][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:39:47,977][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:39:48,481][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:39:49,004][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:39:49,506][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:39:50,009][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:39:50,512][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:39:51,015][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:39:51,517][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:39:52,016][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:39:52,515][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:39:53,016][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:39:53,513][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:39:54,017][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:39:54,516][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:39:55,013][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:39:55,511][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:39:56,013][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:39:56,519][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10443 tokens.
[2025-11-13 01:39:57,193][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 01:39:57,963][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:39:57,965][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:39:57,968][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:39:58,909][__main__][INFO] - Iteration 237 took 53s (32.37% Gen, 65.86% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 55m 35s. Estimated total time: 44h 25m 52s. Time estimates for 10 more iterations: 8m 53s, 100 more iterations: 1h 28m 51s, 500 more iterations: 7h 24m 18s.
[2025-11-13 01:39:58,911][__main__][INFO] - Starting iteration 237.
[2025-11-13 01:39:59,397][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:39:59,398][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:40:03,402][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:40:16,338][__main__][INFO] - Number of regex retries in iteration 237: 1
[2025-11-13 01:40:16,338][__main__][INFO] - agents played in iteration 237 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:40:17,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:40:17,201][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:40:17,224][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:40:17,248][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:40:17,249][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:40:17,249][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:40:17,968][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:40:18,424][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:40:18,928][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:40:19,426][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:40:19,926][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:40:20,424][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:40:20,921][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:40:21,419][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:40:21,917][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:40:22,418][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:40:22,918][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:40:23,419][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:40:23,917][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:40:24,414][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:40:24,917][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:40:25,414][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:40:25,914][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:40:26,414][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:40:26,913][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:40:27,413][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:40:27,914][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:40:28,412][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:40:28,913][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:40:29,421][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:40:29,921][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:40:30,422][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:40:30,951][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:40:31,452][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:40:31,967][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:40:32,466][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:40:32,965][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:40:33,465][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:40:33,964][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:40:34,468][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:40:34,965][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:40:35,461][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:40:35,976][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:40:36,477][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:40:36,981][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:40:37,476][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:40:37,972][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:40:38,475][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:40:38,973][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:40:39,470][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:40:39,969][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:40:40,469][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:40:40,966][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:40:41,462][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:40:41,962][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:40:42,470][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:40:42,970][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:40:43,467][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:40:43,969][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:40:44,465][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:40:44,964][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:40:45,463][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:40:45,963][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:40:46,479][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:40:46,977][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:40:47,480][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:40:47,976][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:40:48,476][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:40:48,970][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:40:49,470][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:40:49,969][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10269 tokens.
[2025-11-13 01:40:50,657][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.40%, ΔTime: 00:00:32
[2025-11-13 01:40:51,406][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:40:51,408][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:40:51,410][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:40:52,319][__main__][INFO] - Iteration 238 took 52s (32.01% Gen, 66.27% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 34m 57s. Estimated total time: 44h 6m 8s. Time estimates for 10 more iterations: 8m 49s, 100 more iterations: 1h 28m 12s, 500 more iterations: 7h 21m 1s.
[2025-11-13 01:40:52,321][__main__][INFO] - Starting iteration 238.
[2025-11-13 01:40:52,798][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:40:52,799][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:41:10,967][__main__][INFO] - Number of regex retries in iteration 238: 0
[2025-11-13 01:41:10,968][__main__][INFO] - agents played in iteration 238 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:41:11,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:41:11,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:41:11,840][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:41:11,863][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:41:11,863][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:41:11,864][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:41:12,556][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:41:13,017][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:41:13,519][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:41:14,017][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:41:14,521][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:41:15,021][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:41:15,525][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:41:16,023][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:41:16,522][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:41:17,022][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:41:17,521][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:41:18,021][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:41:18,517][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:41:19,013][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:41:19,512][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:41:20,010][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:41:20,509][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:41:21,009][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:41:21,507][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:41:22,011][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:41:22,509][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:41:23,007][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:41:23,506][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:41:24,005][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:41:24,506][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:41:25,002][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:41:25,498][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:41:25,999][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:41:26,495][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:41:26,991][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:41:27,489][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:41:27,986][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:41:28,487][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:41:28,986][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:41:29,483][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:41:29,986][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:41:30,485][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:41:30,988][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:41:31,492][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:41:31,988][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:41:32,488][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:41:32,984][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:41:33,484][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:41:33,983][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:41:34,483][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:41:34,988][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:41:35,487][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:41:35,982][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:41:36,492][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:41:36,994][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:41:37,494][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:41:37,992][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:41:38,493][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:41:38,993][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:41:39,491][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:41:39,989][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:41:40,492][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:41:40,990][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:41:41,488][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:41:41,983][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:41:42,477][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:41:42,985][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:41:43,484][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:41:43,982][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:41:44,479][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10263 tokens.
[2025-11-13 01:41:45,166][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.00%, ΔTime: 00:00:32
[2025-11-13 01:41:45,920][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:41:45,922][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:41:45,923][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:41:46,901][__main__][INFO] - Iteration 239 took 54s (33.58% Gen, 64.61% Train). Generation: 18s, Training: 34s. Estimated remaining time: 41h 33m 6s. Estimated total time: 45h 5m 11s. Time estimates for 10 more iterations: 9m 1s, 100 more iterations: 1h 30m 10s, 500 more iterations: 7h 30m 51s.
[2025-11-13 01:41:46,903][__main__][INFO] - Starting iteration 239.
[2025-11-13 01:41:47,370][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:41:47,371][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:42:00,566][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:42:04,452][__main__][INFO] - Number of regex retries in iteration 239: 1
[2025-11-13 01:42:04,453][__main__][INFO] - agents played in iteration 239 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:42:05,248][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:42:05,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:42:05,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:42:05,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:42:05,327][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:42:05,328][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:42:06,030][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:42:06,506][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:42:06,999][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:42:07,501][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:42:08,002][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:42:08,503][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:42:09,004][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:42:09,506][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:42:10,008][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:42:10,510][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:42:11,011][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:42:11,513][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:42:12,016][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:42:12,515][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:42:13,020][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:42:13,521][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:42:14,018][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:42:14,517][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:42:15,019][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:42:15,515][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:42:16,016][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:42:16,511][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:42:17,007][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:42:17,509][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:42:18,009][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:42:18,511][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:42:19,006][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:42:19,503][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:42:20,001][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:42:20,498][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:42:20,995][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:42:21,501][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:42:21,998][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:42:22,504][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:42:23,004][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:42:23,507][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:42:24,010][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:42:24,533][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:42:25,033][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:42:25,530][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:42:26,030][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:42:26,537][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:42:27,037][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:42:27,537][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:42:28,040][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:42:28,538][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:42:29,040][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:42:29,537][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:42:30,035][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:42:30,538][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:42:31,037][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:42:31,536][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:42:32,043][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:42:32,543][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:42:33,037][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:42:33,536][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:42:34,034][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:42:34,541][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:42:35,041][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:42:35,542][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:42:36,041][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:42:36,541][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:42:37,046][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:42:37,546][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:42:38,045][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10464 tokens.
[2025-11-13 01:42:38,744][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.39%, ΔTime: 00:00:32
[2025-11-13 01:42:39,494][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:42:39,496][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:42:39,497][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:42:40,713][__main__][INFO] - Iteration 240 took 53s (32.02% Gen, 65.70% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 54m 9s. Estimated total time: 44h 27m 8s. Time estimates for 10 more iterations: 8m 53s, 100 more iterations: 1h 28m 54s, 500 more iterations: 7h 24m 31s.
[2025-11-13 01:42:40,715][__main__][INFO] - Starting iteration 240.
[2025-11-13 01:42:41,187][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 01:42:41,187][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:42:54,022][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given the values, I aim to maximize my points by focusing on the items where I have the highest per-item value. In this round, both hats and balls have a per-item value of 1 for me, while books are valued at 10 for Bob. Since the values for hats and balls are the same for me, splitting them would be less advantageous due to the risk of them being allocated proportionally if my proposal exceeds the item quantity. 

By proposing to take all the hats, I ensure I get the maximum quantity of hats, which will contribute the most to my points in this round. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:42:59,699][__main__][INFO] - Number of regex retries in iteration 240: 1
[2025-11-13 01:42:59,700][__main__][INFO] - agents played in iteration 240 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:43:00,506][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:43:00,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:43:00,556][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:43:00,579][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:43:00,580][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:43:00,580][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:43:01,326][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:43:01,782][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:43:02,288][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:43:02,787][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:43:03,284][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:43:03,782][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:43:04,283][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:43:04,799][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:43:05,297][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:43:05,793][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:43:06,297][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:43:06,798][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:43:07,308][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:43:07,805][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:43:08,307][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:43:08,821][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:43:09,318][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:43:09,814][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:43:10,316][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:43:10,814][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:43:11,313][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:43:11,813][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:43:12,310][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:43:12,817][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:43:13,319][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:43:13,817][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:43:14,320][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:43:14,818][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:43:15,321][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:43:15,820][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:43:16,317][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:43:16,825][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:43:17,324][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:43:17,823][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:43:18,317][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:43:18,819][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:43:19,317][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:43:19,818][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:43:20,316][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:43:20,823][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:43:21,318][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:43:21,816][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:43:22,318][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:43:22,815][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:43:23,321][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:43:23,822][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:43:24,321][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:43:24,819][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:43:25,319][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:43:25,845][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:43:26,346][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:43:26,843][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:43:27,342][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:43:27,843][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:43:28,339][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:43:28,837][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:43:29,338][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:43:29,838][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:43:30,334][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:43:30,832][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:43:31,330][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:43:31,828][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:43:32,326][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:43:32,825][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:43:33,320][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10382 tokens.
[2025-11-13 01:43:34,010][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-13 01:43:34,762][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:43:34,763][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:43:34,765][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:43:36,963][__main__][INFO] - Iteration 241 took 55s (33.19% Gen, 62.87% Train). Generation: 18s, Training: 35s. Estimated remaining time: 42h 54m 54s. Estimated total time: 46h 28m 49s. Time estimates for 10 more iterations: 9m 17s, 100 more iterations: 1h 32m 57s, 500 more iterations: 7h 44m 48s.
[2025-11-13 01:43:36,965][__main__][INFO] - Starting iteration 241.
[2025-11-13 01:43:37,441][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:43:37,442][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:43:42,310][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:43:55,077][__main__][INFO] - Number of regex retries in iteration 241: 1
[2025-11-13 01:43:55,078][__main__][INFO] - agents played in iteration 241 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:43:55,899][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:43:55,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:43:55,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:43:55,968][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:43:55,968][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:43:55,969][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:43:56,636][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:43:57,092][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:43:57,595][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:43:58,094][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:43:58,613][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:43:59,108][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:43:59,607][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:44:00,105][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:44:00,604][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:44:01,101][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:44:01,600][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:44:02,099][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:44:02,597][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:44:03,097][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:44:03,595][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:44:04,090][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:44:04,586][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:44:05,092][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:44:05,593][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:44:06,093][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:44:06,594][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:44:07,093][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:44:07,590][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:44:08,103][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:44:08,603][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:44:09,114][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:44:09,617][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:44:10,117][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:44:10,619][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:44:11,117][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:44:11,620][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:44:12,118][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:44:12,620][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:44:13,128][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:44:13,626][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:44:14,130][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:44:14,626][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:44:15,125][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:44:15,630][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:44:16,133][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:44:16,635][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:44:17,138][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:44:17,637][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:44:18,136][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:44:18,636][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:44:19,133][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:44:19,637][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:44:20,134][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:44:20,635][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:44:21,134][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:44:21,635][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:44:22,132][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:44:22,630][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:44:23,127][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:44:23,635][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:44:24,134][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:44:24,637][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:44:25,133][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:44:25,633][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:44:26,136][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:44:26,636][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:44:27,133][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:44:27,632][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:44:28,131][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:44:28,631][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10382 tokens.
[2025-11-13 01:44:29,383][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 01:44:30,167][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:44:30,169][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:44:30,170][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:44:31,082][__main__][INFO] - Iteration 242 took 53s (32.88% Gen, 65.42% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 7m 13s. Estimated total time: 44h 42m 2s. Time estimates for 10 more iterations: 8m 56s, 100 more iterations: 1h 29m 24s, 500 more iterations: 7h 27m 0s.
[2025-11-13 01:44:31,084][__main__][INFO] - Starting iteration 242.
[2025-11-13 01:44:31,565][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:44:31,566][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:44:47,430][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:44:48,179][__main__][INFO] - Number of regex retries in iteration 242: 1
[2025-11-13 01:44:48,179][__main__][INFO] - agents played in iteration 242 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:44:48,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:44:48,999][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:44:49,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:44:49,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:44:49,045][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:44:49,046][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:44:49,698][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:44:50,159][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:44:50,663][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:44:51,162][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:44:51,660][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:44:52,159][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:44:52,662][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:44:53,162][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:44:53,661][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:44:54,156][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:44:54,655][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:44:55,151][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:44:55,648][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:44:56,145][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:44:56,643][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:44:57,141][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:44:57,636][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:44:58,131][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:44:58,633][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:44:59,137][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:44:59,640][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:45:00,137][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:45:00,639][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:45:01,141][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:45:01,641][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:45:02,142][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:45:02,640][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:45:03,143][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:45:03,638][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:45:04,136][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:45:04,635][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:45:05,138][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:45:05,639][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:45:06,137][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:45:06,636][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:45:07,144][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:45:07,640][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:45:08,139][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:45:08,641][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:45:09,141][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:45:09,647][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:45:10,146][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:45:10,644][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:45:11,146][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:45:11,644][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:45:12,143][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:45:12,639][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:45:13,138][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:45:13,644][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:45:14,139][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:45:14,636][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:45:15,137][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:45:15,635][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:45:16,135][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:45:16,633][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:45:17,131][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:45:17,642][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:45:18,141][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:45:18,640][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:45:19,146][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:45:19,642][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:45:20,147][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:45:20,648][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:45:21,151][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:45:21,655][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10387 tokens.
[2025-11-13 01:45:22,376][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.37%, ΔTime: 00:00:32
[2025-11-13 01:45:23,134][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:45:23,136][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:45:23,137][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:45:24,017][__main__][INFO] - Iteration 243 took 52s (31.67% Gen, 66.65% Train). Generation: 16s, Training: 34s. Estimated remaining time: 40h 6m 56s. Estimated total time: 43h 42m 38s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 25s, 500 more iterations: 7h 17m 6s.
[2025-11-13 01:45:24,020][__main__][INFO] - Starting iteration 243.
[2025-11-13 01:45:24,494][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:45:24,494][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:45:39,327][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:45:42,960][__main__][INFO] - Number of regex retries in iteration 243: 1
[2025-11-13 01:45:42,961][__main__][INFO] - agents played in iteration 243 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:45:43,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:45:43,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:45:43,801][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:45:43,822][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:45:43,823][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:45:43,824][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:45:44,488][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:45:44,946][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:45:45,448][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:45:45,946][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:45:46,442][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:45:46,939][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:45:47,436][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:45:47,937][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:45:48,437][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:45:48,954][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:45:49,453][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:45:49,953][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:45:50,462][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:45:50,967][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:45:51,464][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:45:51,966][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:45:52,464][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:45:52,962][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:45:53,456][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:45:53,951][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:45:54,453][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:45:54,947][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:45:55,451][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:45:55,945][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:45:56,447][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:45:56,945][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:45:57,441][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:45:57,939][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:45:58,434][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:45:58,933][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:45:59,453][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:45:59,956][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:46:00,455][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:46:00,960][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:46:01,458][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:46:01,960][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:46:02,463][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:46:02,962][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:46:03,466][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:46:03,967][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:46:04,466][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:46:04,970][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:46:05,471][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:46:05,972][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:46:06,473][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:46:07,001][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:46:07,498][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:46:08,001][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:46:08,498][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:46:08,998][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:46:09,498][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:46:09,996][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:46:10,496][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:46:10,997][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:46:11,501][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:46:12,001][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:46:12,499][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:46:12,999][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:46:13,498][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:46:13,997][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:46:14,497][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:46:14,996][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:46:15,498][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:46:15,999][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:46:16,499][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10305 tokens.
[2025-11-13 01:46:17,244][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 01:46:17,990][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:46:17,991][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:46:17,994][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:46:18,901][__main__][INFO] - Iteration 244 took 54s (33.94% Gen, 64.39% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 43m 45s. Estimated total time: 45h 20m 22s. Time estimates for 10 more iterations: 9m 4s, 100 more iterations: 1h 30m 40s, 500 more iterations: 7h 33m 23s.
[2025-11-13 01:46:18,903][__main__][INFO] - Starting iteration 244.
[2025-11-13 01:46:19,384][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:46:19,385][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:46:28,377][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:46:36,079][__main__][INFO] - Number of regex retries in iteration 244: 1
[2025-11-13 01:46:36,080][__main__][INFO] - agents played in iteration 244 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:46:36,873][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:46:36,898][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:46:36,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:46:36,945][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:46:36,945][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:46:36,946][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:46:37,607][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:46:38,058][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:46:38,562][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:46:39,064][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:46:39,567][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:46:40,071][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:46:40,566][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:46:41,066][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:46:41,572][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:46:42,073][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:46:42,572][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:46:43,074][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:46:43,571][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:46:44,070][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:46:44,569][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:46:45,069][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:46:45,569][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:46:46,067][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:46:46,565][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:46:47,066][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:46:47,563][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:46:48,063][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:46:48,561][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:46:49,060][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:46:49,565][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:46:50,064][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:46:50,559][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:46:51,076][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:46:51,577][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:46:52,073][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:46:52,573][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:46:53,069][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:46:53,574][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:46:54,079][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:46:54,581][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:46:55,082][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:46:55,583][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:46:56,081][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:46:56,586][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:46:57,093][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:46:57,595][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:46:58,099][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:46:58,603][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:46:59,105][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:46:59,607][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:47:00,115][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:47:00,618][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:47:01,121][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:47:01,622][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:47:02,123][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:47:02,638][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:47:03,143][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:47:03,646][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:47:04,165][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:47:04,671][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:47:05,173][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:47:05,680][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:47:06,183][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:47:06,690][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:47:07,190][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:47:07,691][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:47:08,195][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:47:08,699][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:47:09,204][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:47:09,706][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10573 tokens.
[2025-11-13 01:47:10,411][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 01:47:11,164][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:47:11,166][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:47:11,167][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:47:12,089][__main__][INFO] - Iteration 245 took 52s (31.68% Gen, 66.57% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 17m 44s. Estimated total time: 43h 55m 15s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 50s, 500 more iterations: 7h 19m 12s.
[2025-11-13 01:47:12,091][__main__][INFO] - Starting iteration 245.
[2025-11-13 01:47:12,575][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:47:12,576][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:47:31,150][__main__][INFO] - Number of regex retries in iteration 245: 0
[2025-11-13 01:47:31,150][__main__][INFO] - agents played in iteration 245 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:47:31,947][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:47:31,971][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:47:31,995][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:47:32,017][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:47:32,018][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:47:32,019][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:47:32,729][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:47:33,188][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:47:33,697][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:47:34,195][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:47:34,694][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:47:35,197][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:47:35,698][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:47:36,197][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:47:36,693][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:47:37,194][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:47:37,693][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:47:38,189][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:47:38,694][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:47:39,194][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:47:39,692][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:47:40,191][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:47:40,689][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:47:41,187][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:47:41,683][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:47:42,182][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:47:42,682][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:47:43,178][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:47:43,677][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:47:44,174][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:47:44,677][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:47:45,174][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:47:45,672][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:47:46,178][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:47:46,675][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:47:47,171][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:47:47,664][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:47:48,161][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:47:48,670][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:47:49,173][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:47:49,674][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:47:50,175][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:47:50,677][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:47:51,182][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:47:51,678][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:47:52,177][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:47:52,683][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:47:53,183][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:47:53,681][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:47:54,190][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:47:54,690][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:47:55,193][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:47:55,697][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:47:56,200][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:47:56,702][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:47:57,198][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:47:57,698][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:47:58,201][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:47:58,702][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:47:59,202][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:47:59,701][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:48:00,201][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:48:00,704][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:48:01,207][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:48:01,707][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:48:02,214][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:48:02,715][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:48:03,220][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:48:03,722][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:48:04,224][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:48:04,730][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10489 tokens.
[2025-11-13 01:48:05,435][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 01:48:06,193][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:48:06,195][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:48:06,196][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:48:07,122][__main__][INFO] - Iteration 246 took 54s (34.05% Gen, 64.25% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 48m 58s. Estimated total time: 45h 27m 23s. Time estimates for 10 more iterations: 9m 5s, 100 more iterations: 1h 30m 54s, 500 more iterations: 7h 34m 33s.
[2025-11-13 01:48:07,124][__main__][INFO] - Starting iteration 246.
[2025-11-13 01:48:07,609][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:48:07,610][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:48:12,756][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:48:25,939][__main__][INFO] - Number of regex retries in iteration 246: 1
[2025-11-13 01:48:25,940][__main__][INFO] - agents played in iteration 246 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:48:26,794][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:48:26,822][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:48:26,848][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:48:26,871][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:48:26,872][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:48:26,873][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:48:27,606][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:48:28,059][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:48:28,577][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:48:29,076][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:48:29,572][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:48:30,069][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:48:30,565][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:48:31,062][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:48:31,562][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:48:32,065][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:48:32,595][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:48:33,094][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:48:33,594][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:48:34,090][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:48:34,588][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:48:35,092][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:48:35,588][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:48:36,088][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:48:36,588][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:48:37,086][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:48:37,584][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:48:38,079][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:48:38,577][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:48:39,087][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:48:39,585][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:48:40,080][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:48:40,577][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:48:41,073][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:48:41,569][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:48:42,069][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:48:42,565][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:48:43,072][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:48:43,572][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:48:44,075][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:48:44,587][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:48:45,087][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:48:45,589][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:48:46,086][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:48:46,587][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:48:47,090][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:48:47,595][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:48:48,099][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:48:48,605][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:48:49,105][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:48:49,608][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:48:50,109][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:48:50,611][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:48:51,118][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:48:51,619][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:48:52,117][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:48:52,621][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:48:53,119][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:48:53,617][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:48:54,119][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:48:54,616][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:48:55,116][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:48:55,617][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:48:56,115][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:48:56,618][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:48:57,117][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:48:57,620][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:48:58,117][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:48:58,616][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:48:59,120][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:48:59,617][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10483 tokens.
[2025-11-13 01:49:00,337][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 01:49:01,095][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:49:01,097][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:49:01,099][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:49:02,043][__main__][INFO] - Iteration 247 took 54s (33.67% Gen, 64.59% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 42m 21s. Estimated total time: 45h 21m 41s. Time estimates for 10 more iterations: 9m 4s, 100 more iterations: 1h 30m 43s, 500 more iterations: 7h 33m 36s.
[2025-11-13 01:49:02,045][__main__][INFO] - Starting iteration 247.
[2025-11-13 01:49:02,511][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:49:02,512][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:49:10,234][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:49:20,589][__main__][INFO] - Number of regex retries in iteration 247: 1
[2025-11-13 01:49:20,590][__main__][INFO] - agents played in iteration 247 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:49:21,461][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:49:21,489][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:49:21,516][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:49:21,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:49:21,540][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:49:21,541][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:49:22,207][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:49:22,677][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:49:23,181][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:49:23,680][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:49:24,189][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:49:24,689][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:49:25,185][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:49:25,680][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:49:26,178][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:49:26,689][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:49:27,189][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:49:27,689][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:49:28,187][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:49:28,681][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:49:29,185][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:49:29,683][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:49:30,180][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:49:30,688][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:49:31,189][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:49:31,684][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:49:32,187][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:49:32,682][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:49:33,185][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:49:33,679][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:49:34,177][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:49:34,675][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:49:35,174][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:49:35,670][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:49:36,175][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:49:36,678][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:49:37,182][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:49:37,683][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:49:38,184][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:49:38,689][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:49:39,193][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:49:39,692][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:49:40,197][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:49:40,699][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:49:41,204][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:49:41,706][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:49:42,205][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:49:42,704][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:49:43,207][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:49:43,708][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:49:44,207][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:49:44,709][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:49:45,213][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:49:45,714][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:49:46,214][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:49:46,714][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:49:47,215][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:49:47,717][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:49:48,217][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:49:48,717][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:49:49,218][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:49:49,719][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:49:50,219][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:49:50,716][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:49:51,217][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:49:51,714][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:49:52,215][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:49:52,710][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:49:53,214][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:49:53,714][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:49:54,216][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10454 tokens.
[2025-11-13 01:49:54,996][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 01:49:55,802][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:49:55,804][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:49:55,806][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:49:56,899][__main__][INFO] - Iteration 248 took 54s (33.24% Gen, 64.75% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 39m 10s. Estimated total time: 45h 19m 25s. Time estimates for 10 more iterations: 9m 3s, 100 more iterations: 1h 30m 38s, 500 more iterations: 7h 33m 14s.
[2025-11-13 01:49:56,902][__main__][INFO] - Starting iteration 248.
[2025-11-13 01:49:57,377][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:49:57,378][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:50:14,101][__main__][INFO] - Number of regex retries in iteration 248: 0
[2025-11-13 01:50:14,101][__main__][INFO] - agents played in iteration 248 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:50:14,874][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:50:14,901][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:50:14,938][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:50:14,962][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:50:14,963][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:50:14,964][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:50:15,611][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:50:16,069][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:50:16,577][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:50:17,083][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:50:17,583][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:50:18,084][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:50:18,581][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:50:19,094][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:50:19,596][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:50:20,094][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:50:20,593][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:50:21,093][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:50:21,604][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:50:22,100][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:50:22,596][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:50:23,097][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:50:23,594][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:50:24,089][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:50:24,594][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:50:25,088][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:50:25,589][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:50:26,083][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:50:26,578][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:50:27,075][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:50:27,570][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:50:28,071][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:50:28,567][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:50:29,066][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:50:29,563][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:50:30,058][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:50:30,556][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:50:31,059][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:50:31,562][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:50:32,065][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:50:32,570][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:50:33,070][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:50:33,571][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:50:34,070][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:50:34,572][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:50:35,083][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:50:35,584][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:50:36,082][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:50:36,582][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:50:37,085][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:50:37,583][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:50:38,081][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:50:38,579][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:50:39,096][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:50:39,598][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:50:40,097][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:50:40,598][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:50:41,103][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:50:41,612][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:50:42,116][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:50:42,642][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:50:43,145][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:50:43,645][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:50:44,141][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:50:44,640][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:50:45,139][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:50:45,638][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:50:46,141][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:50:46,641][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:50:47,140][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:50:47,638][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10500 tokens.
[2025-11-13 01:50:48,397][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.48%, ΔTime: 00:00:32
[2025-11-13 01:50:49,157][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:50:49,159][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:50:49,161][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:50:50,178][__main__][INFO] - Iteration 249 took 52s (31.67% Gen, 66.40% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 18m 55s. Estimated total time: 44h 0m 3s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 0s, 500 more iterations: 7h 20m 0s.
[2025-11-13 01:50:50,180][__main__][INFO] - Starting iteration 249.
[2025-11-13 01:50:50,653][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:50:50,654][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:51:07,036][__main__][INFO] - Number of regex retries in iteration 249: 0
[2025-11-13 01:51:07,037][__main__][INFO] - agents played in iteration 249 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:51:07,901][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:51:07,928][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:51:07,955][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:51:07,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:51:07,978][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:51:07,979][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:51:08,677][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:51:09,139][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:51:09,643][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:51:10,143][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:51:10,642][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:51:11,146][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:51:11,646][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:51:12,141][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:51:12,641][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:51:13,145][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:51:13,644][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:51:14,140][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:51:14,647][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:51:15,148][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:51:15,650][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:51:16,150][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:51:16,650][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:51:17,149][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:51:17,646][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:51:18,145][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:51:18,645][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:51:19,144][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:51:19,644][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:51:20,141][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:51:20,641][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:51:21,139][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:51:21,643][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:51:22,144][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:51:22,647][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:51:23,151][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:51:23,656][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:51:24,157][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:51:24,661][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:51:25,169][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:51:25,669][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:51:26,173][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:51:26,676][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:51:27,180][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:51:27,682][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:51:28,185][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:51:28,687][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:51:29,193][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:51:29,697][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:51:30,199][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:51:30,710][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:51:31,213][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:51:31,731][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:51:32,233][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:51:32,733][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:51:33,242][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:51:33,742][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:51:34,242][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:51:34,743][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:51:35,242][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:51:35,749][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:51:36,255][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:51:36,759][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:51:37,259][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:51:37,761][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:51:38,261][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:51:38,761][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:51:39,265][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:51:39,766][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:51:40,267][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:51:40,768][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10636 tokens.
[2025-11-13 01:51:41,516][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 01:51:42,305][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:51:42,307][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:51:42,309][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:51:43,273][__main__][INFO] - Iteration 250 took 52s (31.14% Gen, 67.03% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 8m 59s. Estimated total time: 43h 51m 0s. Time estimates for 10 more iterations: 8m 46s, 100 more iterations: 1h 27m 42s, 500 more iterations: 7h 18m 30s.
[2025-11-13 01:51:43,275][__main__][INFO] - Starting iteration 250.
[2025-11-13 01:51:43,765][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 01:51:43,765][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:51:48,709][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:52:00,528][__main__][INFO] - Number of regex retries in iteration 250: 1
[2025-11-13 01:52:00,529][__main__][INFO] - agents played in iteration 250 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:52:01,444][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:52:01,467][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:52:01,489][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:52:01,511][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:52:01,511][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:52:01,512][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:52:02,226][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:52:02,679][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:52:03,182][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:52:03,683][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:52:04,184][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:52:04,683][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:52:05,181][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:52:05,677][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:52:06,172][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:52:06,671][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:52:07,167][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:52:07,663][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:52:08,159][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:52:08,656][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:52:09,152][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:52:09,649][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:52:10,145][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:52:10,639][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:52:11,140][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:52:11,635][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:52:12,130][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:52:12,639][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:52:13,134][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:52:13,632][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:52:14,127][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:52:14,623][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:52:15,130][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:52:15,630][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:52:16,131][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:52:16,644][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:52:17,142][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:52:17,644][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:52:18,143][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:52:18,641][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:52:19,146][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:52:19,645][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:52:20,145][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:52:20,645][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:52:21,165][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:52:21,668][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:52:22,168][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:52:22,672][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:52:23,180][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:52:23,680][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:52:24,183][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:52:24,693][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:52:25,195][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:52:25,706][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:52:26,206][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:52:26,705][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:52:27,213][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:52:27,712][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:52:28,223][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:52:28,722][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:52:29,221][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:52:29,723][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:52:30,222][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:52:30,720][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:52:31,221][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:52:31,719][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:52:32,235][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:52:32,731][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:52:33,232][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:52:33,731][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:52:34,230][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10391 tokens.
[2025-11-13 01:52:34,954][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 01:52:35,726][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:52:35,728][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:52:35,730][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:52:37,892][__main__][INFO] - Iteration 251 took 54s (30.97% Gen, 65.03% Train). Generation: 16s, Training: 35s. Estimated remaining time: 41h 23m 26s. Estimated total time: 45h 6m 22s. Time estimates for 10 more iterations: 9m 1s, 100 more iterations: 1h 30m 12s, 500 more iterations: 7h 31m 3s.
[2025-11-13 01:52:37,894][__main__][INFO] - Starting iteration 251.
[2025-11-13 01:52:38,377][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 01:52:38,378][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:52:43,014][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:52:55,580][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:52:56,380][__main__][INFO] - Number of regex retries in iteration 251: 2
[2025-11-13 01:52:56,381][__main__][INFO] - agents played in iteration 251 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:52:57,165][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:52:57,190][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:52:57,215][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:52:57,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:52:57,237][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:52:57,238][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:52:57,882][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:52:58,337][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:52:58,839][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:52:59,334][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:52:59,845][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:53:00,343][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:53:00,840][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:53:01,349][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:53:01,847][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:53:02,340][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:53:02,837][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:53:03,332][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:53:03,838][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:53:04,334][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:53:04,829][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:53:05,327][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:53:05,824][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:53:06,318][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:53:06,819][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:53:07,317][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:53:07,822][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:53:08,320][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:53:08,827][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:53:09,361][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:53:09,864][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:53:10,366][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:53:10,870][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:53:11,377][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:53:11,888][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:53:12,391][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:53:12,893][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:53:13,400][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:53:13,904][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:53:14,403][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:53:14,912][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:53:15,410][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:53:15,922][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:53:16,420][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:53:16,919][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:53:17,427][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:53:17,925][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:53:18,438][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:53:18,937][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:53:19,437][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:53:19,953][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:53:20,452][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:53:20,953][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:53:21,449][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:53:21,948][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:53:22,458][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:53:22,957][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:53:23,452][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:53:23,948][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:53:24,444][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:53:24,950][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:53:25,448][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:53:25,945][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:53:26,460][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:53:26,963][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:53:27,468][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:53:27,978][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:53:28,478][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:53:28,980][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:53:29,481][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:53:29,981][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10416 tokens.
[2025-11-13 01:53:30,725][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.36%, ΔTime: 00:00:32
[2025-11-13 01:53:31,496][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:53:31,497][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:53:31,499][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:53:32,393][__main__][INFO] - Iteration 252 took 54s (33.33% Gen, 65.01% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 16m 58s. Estimated total time: 45h 0m 49s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 1s, 500 more iterations: 7h 30m 8s.
[2025-11-13 01:53:32,395][__main__][INFO] - Starting iteration 252.
[2025-11-13 01:53:32,875][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 01:53:32,876][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:53:49,278][__main__][INFO] - Number of regex retries in iteration 252: 0
[2025-11-13 01:53:49,279][__main__][INFO] - agents played in iteration 252 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:53:50,057][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:53:50,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:53:50,110][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:53:50,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:53:50,134][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:53:50,135][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:53:50,782][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:53:51,234][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:53:51,740][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:53:52,235][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:53:52,734][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:53:53,235][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:53:53,729][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:53:54,226][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:53:54,721][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:53:55,221][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:53:55,860][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:53:56,369][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:53:56,873][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:53:57,371][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:53:57,874][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:53:58,370][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:53:58,866][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:53:59,377][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:53:59,895][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:54:00,401][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:54:00,900][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:54:01,404][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:54:01,910][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:54:02,412][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:54:02,913][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:54:03,415][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:54:03,916][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:54:04,417][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:54:04,918][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:54:05,423][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:54:05,927][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:54:06,429][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:54:06,931][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:54:07,440][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:54:07,943][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:54:08,445][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:54:08,947][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:54:09,449][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:54:09,964][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:54:10,465][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:54:10,968][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:54:11,470][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:54:11,970][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:54:12,475][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:54:12,972][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:54:13,469][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:54:13,979][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:54:14,477][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:54:14,974][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:54:15,475][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:54:15,971][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:54:16,471][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:54:16,968][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:54:17,470][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:54:17,968][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:54:18,468][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:54:18,965][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:54:19,468][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:54:19,976][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:54:20,488][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:54:20,991][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:54:21,495][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:54:21,998][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:54:22,501][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:54:23,009][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10447 tokens.
[2025-11-13 01:54:23,764][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 01:54:24,520][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:54:24,521][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:54:24,523][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:54:25,415][__main__][INFO] - Iteration 253 took 52s (31.22% Gen, 67.08% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 2m 18s. Estimated total time: 43h 47m 1s. Time estimates for 10 more iterations: 8m 45s, 100 more iterations: 1h 27m 34s, 500 more iterations: 7h 17m 50s.
[2025-11-13 01:54:25,418][__main__][INFO] - Starting iteration 253.
[2025-11-13 01:54:25,904][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 01:54:25,905][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:54:43,046][__main__][INFO] - Number of regex retries in iteration 253: 0
[2025-11-13 01:54:43,047][__main__][INFO] - agents played in iteration 253 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:54:43,840][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:54:43,863][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:54:43,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:54:43,909][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:54:43,909][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:54:43,911][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:54:44,575][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:54:45,033][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:54:45,538][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:54:46,037][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:54:46,536][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:54:47,048][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:54:47,551][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:54:48,058][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:54:48,556][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:54:49,055][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:54:49,568][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:54:50,066][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:54:50,564][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:54:51,062][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:54:51,560][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:54:52,076][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:54:52,575][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:54:53,071][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:54:53,571][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:54:54,068][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:54:54,578][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:54:55,081][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:54:55,580][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:54:56,083][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:54:56,584][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:54:57,082][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:54:57,585][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:54:58,089][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:54:58,595][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:54:59,099][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:54:59,601][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:55:00,103][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:55:00,604][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:55:01,108][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:55:01,611][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:55:02,109][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:55:02,613][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:55:03,114][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:55:03,616][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:55:04,117][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:55:04,617][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:55:05,121][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:55:05,618][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:55:06,117][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:55:06,618][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:55:07,118][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:55:07,618][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:55:08,119][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:55:08,618][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:55:09,117][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:55:09,613][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:55:10,113][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:55:10,609][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:55:11,106][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:55:11,605][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:55:12,104][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:55:12,603][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:55:13,107][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:55:13,607][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:55:14,110][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:55:14,612][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:55:15,115][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:55:15,619][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:55:16,120][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:55:16,618][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10429 tokens.
[2025-11-13 01:55:17,346][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 01:55:18,116][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:55:18,118][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:55:18,120][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:55:19,085][__main__][INFO] - Iteration 254 took 53s (32.23% Gen, 65.95% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 33m 26s. Estimated total time: 44h 19m 4s. Time estimates for 10 more iterations: 8m 51s, 100 more iterations: 1h 28m 38s, 500 more iterations: 7h 23m 10s.
[2025-11-13 01:55:19,088][__main__][INFO] - Starting iteration 254.
[2025-11-13 01:55:19,580][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 01:55:19,580][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:55:36,385][__main__][INFO] - Number of regex retries in iteration 254: 0
[2025-11-13 01:55:36,385][__main__][INFO] - agents played in iteration 254 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:55:37,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:55:37,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:55:37,261][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:55:37,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:55:37,284][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:55:37,285][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:55:37,940][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:55:38,394][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:55:38,896][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:55:39,394][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:55:39,897][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:55:40,394][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:55:40,901][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:55:41,399][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:55:41,897][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:55:42,393][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:55:42,892][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:55:43,397][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:55:43,898][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:55:44,396][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:55:44,901][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:55:45,399][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:55:45,897][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:55:46,399][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:55:46,898][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:55:47,419][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:55:47,925][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:55:48,431][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:55:48,937][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:55:49,438][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:55:49,940][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:55:50,441][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:55:50,943][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:55:51,444][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:55:51,946][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:55:52,447][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:55:52,955][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:55:53,457][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:55:53,958][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:55:54,472][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:55:54,975][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:55:55,497][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:55:56,000][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:55:56,504][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:55:57,006][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:55:57,507][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:55:58,013][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:55:58,514][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:55:59,014][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:55:59,511][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:56:00,012][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:56:00,511][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:56:01,013][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:56:01,512][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:56:02,014][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:56:02,511][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:56:03,010][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:56:03,506][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:56:04,001][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:56:04,505][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:56:05,008][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:56:05,509][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:56:06,011][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:56:06,516][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:56:07,019][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:56:07,518][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:56:08,019][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:56:08,523][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:56:09,023][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:56:09,522][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:56:10,021][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10438 tokens.
[2025-11-13 01:56:10,766][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.45%, ΔTime: 00:00:32
[2025-11-13 01:56:11,682][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:56:11,684][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:56:11,699][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:56:12,931][__main__][INFO] - Iteration 255 took 53s (31.50% Gen, 66.19% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 41m 4s. Estimated total time: 44h 27m 35s. Time estimates for 10 more iterations: 8m 53s, 100 more iterations: 1h 28m 55s, 500 more iterations: 7h 24m 35s.
[2025-11-13 01:56:12,933][__main__][INFO] - Starting iteration 255.
[2025-11-13 01:56:13,417][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 01:56:13,418][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:56:31,010][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:56:31,831][__main__][INFO] - Number of regex retries in iteration 255: 1
[2025-11-13 01:56:31,831][__main__][INFO] - agents played in iteration 255 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:56:32,681][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:56:32,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:56:32,730][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:56:32,752][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:56:32,753][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:56:32,754][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:56:33,394][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:56:33,864][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:56:34,370][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:56:34,865][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:56:35,362][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:56:35,861][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:56:36,367][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:56:36,866][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:56:37,364][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:56:37,873][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:56:38,372][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:56:38,870][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:56:39,365][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:56:39,865][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:56:40,364][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:56:40,863][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:56:41,364][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:56:41,871][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:56:42,389][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:56:42,907][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:56:43,407][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:56:43,915][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:56:44,418][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:56:44,923][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:56:45,432][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:56:45,938][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:56:46,438][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:56:46,942][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:56:47,443][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:56:47,946][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:56:48,448][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:56:48,946][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:56:49,447][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:56:49,947][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:56:50,449][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:56:50,952][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:56:51,449][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:56:51,952][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:56:52,450][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:56:52,949][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:56:53,449][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:56:53,947][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:56:54,445][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:56:54,944][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:56:55,440][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:56:55,938][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:56:56,438][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:56:56,938][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:56:57,439][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:56:57,942][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:56:58,440][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:56:58,940][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:56:59,442][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:56:59,943][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:57:00,446][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:57:00,947][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:57:01,447][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:57:01,945][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:57:02,446][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:57:02,943][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:57:03,444][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:57:03,943][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:57:04,440][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:57:04,939][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:57:05,439][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10414 tokens.
[2025-11-13 01:57:06,137][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.36%, ΔTime: 00:00:32
[2025-11-13 01:57:06,935][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:57:06,937][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:57:06,939][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:57:07,861][__main__][INFO] - Iteration 256 took 54s (33.82% Gen, 64.48% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 34m 48s. Estimated total time: 45h 22m 14s. Time estimates for 10 more iterations: 9m 4s, 100 more iterations: 1h 30m 44s, 500 more iterations: 7h 33m 42s.
[2025-11-13 01:57:07,863][__main__][INFO] - Starting iteration 256.
[2025-11-13 01:57:08,350][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 01:57:08,351][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:57:17,728][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 30 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 01:57:24,106][__main__][INFO] - Number of regex retries in iteration 256: 1
[2025-11-13 01:57:24,107][__main__][INFO] - agents played in iteration 256 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:57:24,888][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:57:24,915][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:57:24,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:57:24,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:57:24,964][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:57:24,965][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:57:25,639][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:57:26,095][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:57:26,609][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:57:27,107][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:57:27,603][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:57:28,099][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:57:28,602][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:57:29,098][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:57:29,600][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:57:30,102][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:57:30,604][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:57:31,104][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:57:31,603][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:57:32,104][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:57:32,607][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:57:33,109][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:57:33,611][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:57:34,113][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:57:34,619][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:57:35,119][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:57:35,621][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:57:36,121][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:57:36,620][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:57:37,122][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:57:37,622][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:57:38,124][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:57:38,626][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:57:39,125][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:57:39,624][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:57:40,125][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:57:40,627][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:57:41,131][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:57:41,629][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:57:42,132][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:57:42,637][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:57:43,137][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:57:43,638][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:57:44,139][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:57:44,640][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:57:45,143][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:57:45,643][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:57:46,143][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:57:46,644][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:57:47,142][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:57:47,640][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:57:48,143][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:57:48,643][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:57:49,142][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:57:49,645][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:57:50,150][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:57:50,652][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:57:51,157][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:57:51,661][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:57:52,162][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:57:52,663][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:57:53,175][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:57:53,677][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:57:54,186][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:57:54,684][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:57:55,182][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:57:55,685][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:57:56,186][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:57:56,687][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:57:57,190][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:57:57,692][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10418 tokens.
[2025-11-13 01:57:58,397][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 01:57:59,174][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:57:59,175][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:57:59,177][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:58:00,138][__main__][INFO] - Iteration 257 took 51s (30.42% Gen, 67.72% Train). Generation: 15s, Training: 35s. Estimated remaining time: 39h 21m 5s. Estimated total time: 43h 9m 23s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 18s, 500 more iterations: 7h 11m 33s.
[2025-11-13 01:58:00,140][__main__][INFO] - Starting iteration 257.
[2025-11-13 01:58:00,616][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 01:58:00,616][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:58:17,334][__main__][INFO] - Number of regex retries in iteration 257: 0
[2025-11-13 01:58:17,335][__main__][INFO] - agents played in iteration 257 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:58:18,107][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:58:18,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:58:18,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:58:18,178][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:58:18,178][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:58:18,179][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:58:18,828][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:58:19,282][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:58:19,794][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:58:20,294][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:58:20,795][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:58:21,294][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:58:21,794][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:58:22,296][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:58:22,795][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:58:23,296][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:58:23,794][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:58:24,293][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:58:24,818][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:58:25,317][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:58:25,819][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:58:26,318][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:58:26,819][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:58:27,324][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:58:27,822][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:58:28,321][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:58:28,823][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:58:29,320][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:58:29,823][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:58:30,322][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:58:30,820][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:58:31,321][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:58:31,824][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:58:32,323][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:58:32,826][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:58:33,326][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:58:33,830][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:58:34,332][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:58:34,836][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:58:35,339][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:58:35,845][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:58:36,346][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:58:36,845][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:58:37,344][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:58:37,843][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:58:38,343][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:58:38,842][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:58:39,344][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:58:39,841][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:58:40,340][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:58:40,838][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:58:41,340][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:58:41,839][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:58:42,336][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:58:42,838][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:58:43,349][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:58:43,856][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:58:44,362][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:58:44,866][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:58:45,368][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:58:45,882][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:58:46,387][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:58:46,889][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:58:47,394][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:58:47,895][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:58:48,396][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:58:48,896][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:58:49,396][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:58:49,914][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:58:50,412][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:58:50,910][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10464 tokens.
[2025-11-13 01:58:51,661][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 01:58:52,431][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:58:52,433][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:58:52,434][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:58:53,443][__main__][INFO] - Iteration 258 took 52s (31.65% Gen, 66.44% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 12m 12s. Estimated total time: 44h 1m 24s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 2s, 500 more iterations: 7h 20m 14s.
[2025-11-13 01:58:53,445][__main__][INFO] - Starting iteration 258.
[2025-11-13 01:58:53,933][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 01:58:53,933][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 01:59:11,843][__main__][INFO] - Number of regex retries in iteration 258: 0
[2025-11-13 01:59:11,843][__main__][INFO] - agents played in iteration 258 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 01:59:12,619][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:59:12,644][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:59:12,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:59:12,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 01:59:12,692][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 01:59:12,693][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 01:59:13,347][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 01:59:13,800][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 01:59:14,303][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 01:59:14,816][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 01:59:15,317][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 01:59:15,814][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 01:59:16,311][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 01:59:16,806][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 01:59:17,314][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 01:59:17,818][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 01:59:18,321][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 01:59:18,828][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 01:59:19,330][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 01:59:19,832][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 01:59:20,333][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 01:59:20,832][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 01:59:21,342][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 01:59:21,846][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 01:59:22,344][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 01:59:22,848][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 01:59:23,345][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 01:59:23,847][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 01:59:24,363][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 01:59:24,866][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 01:59:25,367][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 01:59:25,866][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 01:59:26,367][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 01:59:26,872][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 01:59:27,375][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 01:59:27,883][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 01:59:28,389][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 01:59:28,891][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 01:59:29,400][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 01:59:29,899][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 01:59:30,412][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 01:59:30,922][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 01:59:31,424][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 01:59:31,926][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 01:59:32,428][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 01:59:32,930][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 01:59:33,434][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 01:59:33,937][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 01:59:34,441][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 01:59:34,946][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 01:59:35,447][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 01:59:35,951][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 01:59:36,455][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 01:59:36,959][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 01:59:37,462][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 01:59:37,965][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 01:59:38,467][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 01:59:38,968][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 01:59:39,467][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 01:59:39,968][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 01:59:40,469][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 01:59:40,969][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 01:59:41,467][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 01:59:41,972][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 01:59:42,470][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 01:59:42,971][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 01:59:43,473][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 01:59:43,981][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 01:59:44,482][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 01:59:45,003][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 01:59:45,505][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10519 tokens.
[2025-11-13 01:59:46,225][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.43%, ΔTime: 00:00:32
[2025-11-13 01:59:46,995][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 01:59:46,997][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 01:59:46,999][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 01:59:47,978][__main__][INFO] - Iteration 259 took 54s (33.14% Gen, 65.05% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 12m 10s. Estimated total time: 45h 2m 16s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 4s, 500 more iterations: 7h 30m 22s.
[2025-11-13 01:59:47,980][__main__][INFO] - Starting iteration 259.
[2025-11-13 01:59:48,468][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 01:59:48,469][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:00:06,378][__main__][INFO] - Number of regex retries in iteration 259: 0
[2025-11-13 02:00:06,379][__main__][INFO] - agents played in iteration 259 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:00:07,277][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:00:07,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:00:07,329][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:00:07,351][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:00:07,352][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:00:07,353][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:00:08,052][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:00:08,509][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:00:09,018][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:00:09,521][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:00:10,027][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:00:10,532][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:00:11,032][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:00:11,536][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:00:12,037][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:00:12,539][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:00:13,040][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:00:13,541][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:00:14,048][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:00:14,551][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:00:15,052][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:00:15,556][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:00:16,057][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:00:16,561][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:00:17,061][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:00:17,561][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:00:18,063][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:00:18,560][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:00:19,059][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:00:19,558][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:00:20,063][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:00:20,566][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:00:21,072][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:00:21,575][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:00:22,078][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:00:22,580][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:00:23,085][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:00:23,588][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:00:24,088][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:00:24,599][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:00:25,100][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:00:25,604][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:00:26,109][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:00:26,611][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:00:27,114][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:00:27,615][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:00:28,113][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:00:28,630][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:00:29,131][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:00:29,630][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:00:30,142][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:00:30,644][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:00:31,146][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:00:31,646][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:00:32,151][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:00:32,654][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:00:33,159][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:00:33,662][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:00:34,162][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:00:34,666][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:00:35,166][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:00:35,667][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:00:36,166][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:00:36,667][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:00:37,167][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:00:37,670][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:00:38,170][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:00:38,668][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:00:39,173][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:00:39,670][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:00:40,170][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10494 tokens.
[2025-11-13 02:00:40,896][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 02:00:41,648][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:00:41,650][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:00:41,651][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:00:42,585][__main__][INFO] - Iteration 260 took 54s (33.10% Gen, 65.18% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 14m 51s. Estimated total time: 45h 5m 52s. Time estimates for 10 more iterations: 9m 1s, 100 more iterations: 1h 30m 11s, 500 more iterations: 7h 30m 58s.
[2025-11-13 02:00:42,588][__main__][INFO] - Starting iteration 260.
[2025-11-13 02:00:43,079][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 02:00:43,079][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:01:01,257][__main__][INFO] - Number of regex retries in iteration 260: 0
[2025-11-13 02:01:01,257][__main__][INFO] - agents played in iteration 260 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:01:02,149][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:01:02,172][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:01:02,196][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:01:02,218][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:01:02,219][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:01:02,219][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:01:02,923][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:01:03,380][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:01:03,887][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:01:04,388][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:01:04,889][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:01:05,391][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:01:05,891][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:01:06,390][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:01:06,897][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:01:07,398][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:01:07,897][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:01:08,401][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:01:08,902][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:01:09,398][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:01:09,899][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:01:10,395][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:01:10,901][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:01:11,399][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:01:11,897][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:01:12,398][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:01:12,897][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:01:13,398][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:01:13,899][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:01:14,399][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:01:14,902][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:01:15,402][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:01:15,903][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:01:16,402][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:01:16,907][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:01:17,416][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:01:17,916][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:01:18,417][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:01:18,925][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:01:19,430][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:01:19,943][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:01:20,443][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:01:20,946][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:01:21,448][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:01:21,948][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:01:22,451][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:01:22,950][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:01:23,447][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:01:23,953][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:01:24,455][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:01:24,958][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:01:25,466][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:01:25,969][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:01:26,471][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:01:26,970][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:01:27,472][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:01:27,972][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:01:28,472][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:01:28,975][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:01:29,477][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:01:29,979][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:01:30,484][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:01:30,983][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:01:31,479][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:01:31,981][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:01:32,481][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:01:32,980][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:01:33,480][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:01:33,977][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:01:34,479][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:01:34,977][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10475 tokens.
[2025-11-13 02:01:36,057][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 02:01:36,839][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:01:36,840][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:01:36,842][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:01:38,761][__main__][INFO] - Iteration 261 took 55s (32.65% Gen, 63.91% Train). Generation: 18s, Training: 35s. Estimated remaining time: 42h 32m 12s. Estimated total time: 46h 24m 9s. Time estimates for 10 more iterations: 9m 16s, 100 more iterations: 1h 32m 48s, 500 more iterations: 7h 44m 1s.
[2025-11-13 02:01:38,763][__main__][INFO] - Starting iteration 261.
[2025-11-13 02:01:39,232][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:01:39,232][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:01:42,972][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:01:55,749][__main__][INFO] - Number of regex retries in iteration 261: 1
[2025-11-13 02:01:55,750][__main__][INFO] - agents played in iteration 261 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:01:56,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:01:56,597][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:01:56,620][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:01:56,642][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:01:56,643][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:01:56,644][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:01:57,383][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:01:57,840][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:01:58,346][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:01:58,847][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:01:59,362][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:01:59,866][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:02:00,364][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:02:00,866][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:02:01,364][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:02:01,862][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:02:02,359][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:02:02,859][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:02:03,372][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:02:03,870][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:02:04,367][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:02:04,865][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:02:05,362][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:02:05,867][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:02:06,366][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:02:06,865][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:02:07,374][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:02:07,878][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:02:08,384][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:02:08,884][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:02:09,385][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:02:09,890][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:02:10,400][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:02:10,903][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:02:11,406][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:02:11,906][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:02:12,410][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:02:12,913][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:02:13,417][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:02:13,920][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:02:14,420][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:02:14,920][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:02:15,420][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:02:15,920][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:02:16,415][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:02:16,912][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:02:17,415][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:02:17,918][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:02:18,422][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:02:18,925][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:02:19,426][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:02:19,924][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:02:20,429][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:02:20,928][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:02:21,431][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:02:21,931][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:02:22,428][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:02:22,930][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:02:23,426][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:02:23,925][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:02:24,424][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:02:24,919][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:02:25,418][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:02:25,919][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:02:26,419][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:02:26,918][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:02:27,415][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:02:27,913][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:02:28,412][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:02:28,913][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:02:29,412][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10426 tokens.
[2025-11-13 02:02:30,112][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 02:02:30,910][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:02:30,912][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:02:30,913][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:02:31,906][__main__][INFO] - Iteration 262 took 52s (31.36% Gen, 66.76% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 0m 53s. Estimated total time: 43h 53m 43s. Time estimates for 10 more iterations: 8m 46s, 100 more iterations: 1h 27m 47s, 500 more iterations: 7h 18m 57s.
[2025-11-13 02:02:31,908][__main__][INFO] - Starting iteration 262.
[2025-11-13 02:02:32,407][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:02:32,408][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:02:38,173][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:02:50,965][__main__][INFO] - Number of regex retries in iteration 262: 1
[2025-11-13 02:02:50,965][__main__][INFO] - agents played in iteration 262 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:02:51,792][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:02:51,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:02:51,842][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:02:51,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:02:51,865][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:02:51,866][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:02:52,591][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:02:53,050][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:02:53,557][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:02:54,060][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:02:54,561][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:02:55,064][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:02:55,560][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:02:56,071][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:02:56,569][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:02:57,070][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:02:57,583][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:02:58,083][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:02:58,584][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:02:59,093][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:02:59,592][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:03:00,093][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:03:00,594][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:03:01,096][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:03:01,602][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:03:02,103][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:03:02,602][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:03:03,111][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:03:03,610][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:03:04,111][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:03:04,613][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:03:05,115][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:03:05,622][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:03:06,124][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:03:06,625][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:03:07,125][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:03:07,622][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:03:08,122][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:03:08,623][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:03:09,120][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:03:09,620][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:03:10,118][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:03:10,615][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:03:11,118][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:03:11,623][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:03:12,124][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:03:12,629][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:03:13,128][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:03:13,629][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:03:14,129][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:03:14,626][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:03:15,125][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:03:15,623][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:03:16,122][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:03:16,620][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:03:17,120][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:03:17,623][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:03:18,123][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:03:18,622][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:03:19,118][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:03:19,621][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:03:20,121][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:03:20,620][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:03:21,118][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:03:21,619][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:03:22,120][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:03:22,620][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:03:23,117][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:03:23,616][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:03:24,112][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:03:24,611][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10379 tokens.
[2025-11-13 02:03:25,300][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 02:03:26,074][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:03:26,076][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:03:26,078][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:03:27,071][__main__][INFO] - Iteration 263 took 54s (33.95% Gen, 64.23% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 39m 27s. Estimated total time: 45h 33m 13s. Time estimates for 10 more iterations: 9m 6s, 100 more iterations: 1h 31m 6s, 500 more iterations: 7h 35m 32s.
[2025-11-13 02:03:27,073][__main__][INFO] - Starting iteration 263.
[2025-11-13 02:03:27,554][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:03:27,555][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:03:47,187][__main__][INFO] - Number of regex retries in iteration 263: 0
[2025-11-13 02:03:47,188][__main__][INFO] - agents played in iteration 263 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:03:48,056][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:03:48,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:03:48,112][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:03:48,135][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:03:48,135][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:03:48,137][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:03:48,802][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:03:49,263][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:03:49,770][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:03:50,273][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:03:50,787][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:03:51,285][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:03:51,794][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:03:52,295][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:03:52,797][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:03:53,296][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:03:53,799][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:03:54,303][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:03:54,807][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:03:55,308][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:03:55,821][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:03:56,326][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:03:56,830][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:03:57,335][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:03:57,838][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:03:58,342][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:03:58,842][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:03:59,341][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:03:59,842][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:04:00,343][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:04:00,841][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:04:01,348][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:04:01,847][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:04:02,351][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:04:02,850][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:04:03,348][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:04:03,851][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:04:04,354][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:04:04,857][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:04:05,361][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:04:05,861][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:04:06,365][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:04:06,869][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:04:07,372][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:04:07,870][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:04:08,370][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:04:08,882][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:04:09,385][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:04:09,888][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:04:10,394][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:04:10,895][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:04:11,399][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:04:11,903][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:04:12,403][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:04:12,918][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:04:13,424][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:04:13,930][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:04:14,432][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:04:14,936][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:04:15,436][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:04:15,934][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:04:16,431][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:04:16,928][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:04:17,426][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:04:17,923][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:04:18,422][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:04:18,920][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:04:19,424][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:04:19,922][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:04:20,424][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:04:20,924][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10278 tokens.
[2025-11-13 02:04:21,663][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 02:04:22,443][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:04:22,445][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:04:22,446][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:04:23,401][__main__][INFO] - Iteration 264 took 55s (35.15% Gen, 63.13% Train). Generation: 19s, Training: 35s. Estimated remaining time: 42h 37m 39s. Estimated total time: 46h 32m 21s. Time estimates for 10 more iterations: 9m 18s, 100 more iterations: 1h 33m 4s, 500 more iterations: 7h 45m 23s.
[2025-11-13 02:04:23,403][__main__][INFO] - Starting iteration 264.
[2025-11-13 02:04:23,894][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:04:23,894][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:04:42,342][__main__][INFO] - Number of regex retries in iteration 264: 0
[2025-11-13 02:04:42,343][__main__][INFO] - agents played in iteration 264 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:04:43,193][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:04:43,216][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:04:43,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:04:43,259][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:04:43,260][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:04:43,260][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:04:43,932][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:04:44,397][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:04:44,903][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:04:45,402][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:04:45,901][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:04:46,405][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:04:46,921][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:04:47,420][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:04:47,922][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:04:48,422][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:04:48,924][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:04:49,426][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:04:49,925][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:04:50,426][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:04:50,934][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:04:51,439][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:04:51,941][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:04:52,443][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:04:52,943][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:04:53,444][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:04:53,942][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:04:54,440][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:04:54,942][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:04:55,443][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:04:55,942][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:04:56,446][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:04:56,947][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:04:57,449][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:04:57,952][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:04:58,454][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:04:58,957][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:04:59,463][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:04:59,972][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:05:00,468][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:05:00,967][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:05:01,473][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:05:01,972][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:05:02,474][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:05:02,983][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:05:03,483][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:05:03,996][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:05:04,498][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:05:05,000][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:05:05,502][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:05:06,003][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:05:06,515][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:05:07,017][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:05:07,514][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:05:08,017][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:05:08,516][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:05:09,018][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:05:09,513][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:05:10,013][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:05:10,518][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:05:11,016][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:05:11,513][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:05:12,026][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:05:12,525][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:05:13,024][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:05:13,521][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:05:14,019][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:05:14,533][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:05:15,032][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:05:15,528][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:05:16,035][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10394 tokens.
[2025-11-13 02:05:16,721][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 02:05:17,478][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:05:17,479][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:05:17,481][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:05:18,468][__main__][INFO] - Iteration 265 took 54s (33.80% Gen, 64.39% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 33m 9s. Estimated total time: 45h 28m 46s. Time estimates for 10 more iterations: 9m 5s, 100 more iterations: 1h 30m 57s, 500 more iterations: 7h 34m 47s.
[2025-11-13 02:05:18,471][__main__][INFO] - Starting iteration 265.
[2025-11-13 02:05:18,943][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:05:18,943][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:05:23,684][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:05:36,972][__main__][INFO] - Number of regex retries in iteration 265: 1
[2025-11-13 02:05:36,973][__main__][INFO] - agents played in iteration 265 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:05:37,840][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:05:37,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:05:37,893][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:05:37,916][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:05:37,916][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:05:37,917][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:05:38,601][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:05:39,052][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:05:39,561][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:05:40,064][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:05:40,564][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:05:41,068][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:05:41,570][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:05:42,070][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:05:42,574][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:05:43,081][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:05:43,586][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:05:44,088][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:05:44,587][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:05:45,086][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:05:45,586][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:05:46,096][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:05:46,595][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:05:47,095][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:05:47,611][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:05:48,111][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:05:48,624][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:05:49,126][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:05:49,625][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:05:50,140][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:05:50,638][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:05:51,140][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:05:51,632][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:05:52,134][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:05:52,644][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:05:53,145][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:05:53,648][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:05:54,146][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:05:54,645][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:05:55,146][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:05:55,645][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:05:56,146][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:05:56,647][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:05:57,146][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:05:57,647][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:05:58,145][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:05:58,649][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:05:59,177][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:05:59,678][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:06:00,181][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:06:00,683][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:06:01,187][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:06:01,690][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:06:02,186][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:06:02,690][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:06:03,195][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:06:03,701][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:06:04,204][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:06:04,717][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:06:05,218][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:06:05,730][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:06:06,228][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:06:06,728][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:06:07,228][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:06:07,724][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:06:08,224][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:06:08,726][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:06:09,226][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:06:09,724][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:06:10,221][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:06:10,719][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10403 tokens.
[2025-11-13 02:06:11,470][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 02:06:12,251][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:06:12,253][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:06:12,255][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:06:13,185][__main__][INFO] - Iteration 266 took 54s (33.24% Gen, 65.04% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 15m 38s. Estimated total time: 45h 12m 9s. Time estimates for 10 more iterations: 9m 2s, 100 more iterations: 1h 30m 24s, 500 more iterations: 7h 32m 1s.
[2025-11-13 02:06:13,188][__main__][INFO] - Starting iteration 266.
[2025-11-13 02:06:13,679][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:06:13,681][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:06:18,876][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:06:30,728][__main__][INFO] - Number of regex retries in iteration 266: 1
[2025-11-13 02:06:30,728][__main__][INFO] - agents played in iteration 266 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:06:31,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:06:31,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:06:31,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:06:31,607][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:06:31,607][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:06:31,608][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:06:32,289][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:06:32,743][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:06:33,252][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:06:33,753][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:06:34,253][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:06:34,759][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:06:35,254][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:06:35,756][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:06:36,257][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:06:36,757][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:06:37,259][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:06:37,756][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:06:38,256][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:06:38,767][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:06:39,260][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:06:39,761][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:06:40,260][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:06:40,761][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:06:41,263][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:06:41,763][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:06:42,263][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:06:42,764][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:06:43,261][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:06:43,760][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:06:44,259][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:06:44,756][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:06:45,260][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:06:45,759][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:06:46,258][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:06:46,757][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:06:47,258][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:06:47,761][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:06:48,262][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:06:48,759][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:06:49,261][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:06:49,758][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:06:50,256][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:06:50,759][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:06:51,257][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:06:51,762][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:06:52,262][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:06:52,764][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:06:53,264][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:06:53,764][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:06:54,264][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:06:54,762][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:06:55,265][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:06:55,921][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:06:56,430][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:06:56,951][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:06:57,454][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:06:57,957][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:06:58,459][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:06:58,959][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:06:59,461][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:06:59,959][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:07:00,457][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:07:00,958][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:07:01,458][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:07:01,958][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:07:02,458][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:07:02,957][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:07:03,461][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:07:03,961][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:07:04,460][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10318 tokens.
[2025-11-13 02:07:05,191][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 02:07:05,967][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:07:05,969][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:07:05,970][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:07:06,883][__main__][INFO] - Iteration 267 took 53s (32.04% Gen, 66.24% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 22m 47s. Estimated total time: 44h 20m 13s. Time estimates for 10 more iterations: 8m 52s, 100 more iterations: 1h 28m 40s, 500 more iterations: 7h 23m 22s.
[2025-11-13 02:07:06,886][__main__][INFO] - Starting iteration 267.
[2025-11-13 02:07:07,387][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:07:07,388][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:07:24,483][__main__][INFO] - Number of regex retries in iteration 267: 0
[2025-11-13 02:07:24,484][__main__][INFO] - agents played in iteration 267 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:07:25,293][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:07:25,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:07:25,339][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:07:25,361][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:07:25,361][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:07:25,362][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:07:26,085][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:07:26,546][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:07:27,065][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:07:27,563][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:07:28,059][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:07:28,555][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:07:29,050][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:07:29,555][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:07:30,054][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:07:30,556][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:07:31,064][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:07:31,564][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:07:32,068][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:07:32,569][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:07:33,067][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:07:33,574][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:07:34,077][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:07:34,575][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:07:35,079][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:07:35,579][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:07:36,077][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:07:36,579][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:07:37,075][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:07:37,577][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:07:38,073][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:07:38,577][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:07:39,075][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:07:39,580][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:07:40,089][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:07:40,591][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:07:41,093][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:07:41,602][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:07:42,106][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:07:42,608][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:07:43,110][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:07:43,613][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:07:44,120][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:07:44,620][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:07:45,122][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:07:45,636][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:07:46,138][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:07:46,646][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:07:47,145][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:07:47,644][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:07:48,144][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:07:48,646][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:07:49,144][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:07:49,642][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:07:50,141][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:07:50,662][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:07:51,163][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:07:51,667][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:07:52,168][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:07:52,668][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:07:53,176][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:07:53,677][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:07:54,179][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:07:54,675][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:07:55,172][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:07:55,675][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:07:56,172][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:07:56,669][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:07:57,167][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:07:57,664][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:07:58,163][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10332 tokens.
[2025-11-13 02:07:58,890][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.35%, ΔTime: 00:00:32
[2025-11-13 02:07:59,694][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:07:59,696][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:07:59,697][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:08:00,689][__main__][INFO] - Iteration 268 took 53s (32.07% Gen, 66.06% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 26m 49s. Estimated total time: 44h 25m 8s. Time estimates for 10 more iterations: 8m 53s, 100 more iterations: 1h 28m 50s, 500 more iterations: 7h 24m 11s.
[2025-11-13 02:08:00,692][__main__][INFO] - Starting iteration 268.
[2025-11-13 02:08:01,188][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:08:01,189][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:08:16,571][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:08:17,387][__main__][INFO] - Number of regex retries in iteration 268: 1
[2025-11-13 02:08:17,387][__main__][INFO] - agents played in iteration 268 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:08:18,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:08:18,245][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:08:18,268][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:08:18,291][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:08:18,292][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:08:18,293][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:08:18,951][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:08:19,414][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:08:19,918][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:08:20,419][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:08:20,915][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:08:21,418][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:08:21,917][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:08:22,417][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:08:22,917][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:08:23,420][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:08:23,922][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:08:24,423][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:08:24,924][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:08:25,426][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:08:25,931][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:08:26,438][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:08:26,939][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:08:27,448][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:08:27,947][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:08:28,468][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:08:28,967][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:08:29,465][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:08:29,966][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:08:30,465][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:08:30,973][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:08:31,473][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:08:31,973][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:08:32,475][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:08:32,975][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:08:33,479][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:08:33,978][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:08:34,481][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:08:34,980][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:08:35,481][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:08:35,982][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:08:36,482][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:08:36,985][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:08:37,488][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:08:37,988][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:08:38,488][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:08:38,990][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:08:39,489][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:08:39,988][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:08:40,491][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:08:40,992][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:08:41,489][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:08:41,986][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:08:42,487][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:08:42,986][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:08:43,485][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:08:43,984][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:08:44,482][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:08:44,982][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:08:45,486][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:08:45,986][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:08:46,483][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:08:46,982][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:08:47,482][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:08:47,981][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:08:48,482][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:08:48,980][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:08:49,486][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:08:49,987][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:08:50,490][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:08:50,991][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10326 tokens.
[2025-11-13 02:08:51,758][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 02:08:52,568][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:08:52,569][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:08:52,571][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:08:53,622][__main__][INFO] - Iteration 269 took 52s (30.89% Gen, 67.10% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 42m 30s. Estimated total time: 43h 41m 42s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 23s, 500 more iterations: 7h 16m 57s.
[2025-11-13 02:08:53,624][__main__][INFO] - Starting iteration 269.
[2025-11-13 02:08:54,105][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:08:54,106][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:08:59,075][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:09:12,412][__main__][INFO] - Number of regex retries in iteration 269: 1
[2025-11-13 02:09:12,413][__main__][INFO] - agents played in iteration 269 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:09:13,219][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:09:13,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:09:13,268][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:09:13,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:09:13,291][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:09:13,292][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:09:13,953][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:09:14,405][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:09:14,908][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:09:15,406][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:09:15,907][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:09:16,408][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:09:16,908][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:09:17,407][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:09:17,908][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:09:18,408][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:09:18,911][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:09:19,420][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:09:19,927][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:09:20,453][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:09:20,959][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:09:21,470][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:09:21,972][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:09:22,495][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:09:22,998][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:09:23,499][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:09:23,996][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:09:24,493][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:09:24,992][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:09:25,488][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:09:25,986][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:09:26,485][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:09:26,986][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:09:27,483][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:09:27,983][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:09:28,479][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:09:28,991][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:09:29,490][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:09:29,990][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:09:30,495][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:09:30,997][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:09:31,497][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:09:31,998][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:09:32,497][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:09:32,997][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:09:33,499][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:09:34,002][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:09:34,503][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:09:35,003][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:09:35,506][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:09:36,005][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:09:36,507][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:09:37,011][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:09:37,511][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:09:38,012][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:09:38,511][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:09:39,011][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:09:39,517][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:09:40,014][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:09:40,514][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:09:41,014][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:09:41,514][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:09:42,015][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:09:42,518][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:09:43,018][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:09:43,519][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:09:44,019][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:09:44,519][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:09:45,023][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:09:45,521][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:09:46,025][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10293 tokens.
[2025-11-13 02:09:46,763][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 02:09:47,518][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:09:47,521][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:09:47,522][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:09:48,463][__main__][INFO] - Iteration 270 took 54s (33.68% Gen, 64.59% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 17m 48s. Estimated total time: 45h 17m 55s. Time estimates for 10 more iterations: 9m 3s, 100 more iterations: 1h 30m 35s, 500 more iterations: 7h 32m 59s.
[2025-11-13 02:09:48,465][__main__][INFO] - Starting iteration 270.
[2025-11-13 02:09:48,948][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 02:09:48,949][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:09:53,242][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:10:06,293][__main__][INFO] - Number of regex retries in iteration 270: 1
[2025-11-13 02:10:06,294][__main__][INFO] - agents played in iteration 270 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:10:07,190][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:10:07,212][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:10:07,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:10:07,256][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:10:07,256][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:10:07,257][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:10:07,926][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:10:08,389][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:10:08,896][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:10:09,397][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:10:09,897][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:10:10,396][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:10:10,902][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:10:11,405][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:10:11,907][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:10:12,412][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:10:12,912][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:10:13,416][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:10:13,918][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:10:14,418][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:10:14,920][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:10:15,420][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:10:15,923][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:10:16,423][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:10:16,922][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:10:17,424][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:10:17,924][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:10:18,423][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:10:18,924][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:10:19,426][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:10:19,925][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:10:20,427][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:10:20,927][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:10:21,429][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:10:21,929][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:10:22,433][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:10:22,938][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:10:23,434][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:10:23,937][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:10:24,435][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:10:24,937][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:10:25,440][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:10:25,938][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:10:26,438][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:10:26,942][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:10:27,443][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:10:27,946][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:10:28,447][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:10:28,945][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:10:29,445][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:10:29,944][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:10:30,445][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:10:30,945][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:10:31,447][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:10:31,960][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:10:32,458][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:10:32,962][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:10:33,462][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:10:33,962][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:10:34,466][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:10:34,965][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:10:35,469][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:10:35,973][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:10:36,471][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:10:36,983][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:10:37,482][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:10:37,983][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:10:38,481][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:10:38,981][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:10:39,482][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:10:39,980][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10301 tokens.
[2025-11-13 02:10:40,712][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 02:10:41,474][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:10:41,476][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:10:41,478][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:10:43,499][__main__][INFO] - Iteration 271 took 54s (31.79% Gen, 64.50% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 26m 31s. Estimated total time: 45h 27m 33s. Time estimates for 10 more iterations: 9m 5s, 100 more iterations: 1h 30m 55s, 500 more iterations: 7h 34m 35s.
[2025-11-13 02:10:43,501][__main__][INFO] - Starting iteration 271.
[2025-11-13 02:10:43,987][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:10:43,988][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:10:49,009][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:10:52,662][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:11:02,344][__main__][INFO] - Number of regex retries in iteration 271: 2
[2025-11-13 02:11:02,345][__main__][INFO] - agents played in iteration 271 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:11:03,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:11:03,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:11:03,205][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:11:03,229][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:11:03,230][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:11:03,231][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:11:03,886][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:11:04,335][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:11:04,844][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:11:05,342][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:11:05,840][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:11:06,342][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:11:06,839][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:11:07,339][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:11:07,837][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:11:08,342][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:11:08,841][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:11:09,342][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:11:09,846][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:11:10,344][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:11:10,842][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:11:11,343][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:11:11,848][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:11:12,348][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:11:12,846][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:11:13,346][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:11:13,845][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:11:14,343][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:11:14,842][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:11:15,345][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:11:15,847][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:11:16,345][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:11:16,845][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:11:17,344][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:11:17,845][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:11:18,342][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:11:18,842][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:11:19,341][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:11:19,839][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:11:20,344][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:11:20,844][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:11:21,344][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:11:21,848][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:11:22,349][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:11:22,846][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:11:23,347][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:11:23,848][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:11:24,347][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:11:24,846][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:11:25,348][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:11:25,850][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:11:26,350][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:11:26,850][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:11:27,349][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:11:27,850][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:11:28,353][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:11:28,852][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:11:29,351][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:11:29,860][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:11:30,360][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:11:30,858][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:11:31,355][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:11:31,854][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:11:32,350][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:11:32,847][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:11:33,350][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:11:33,848][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:11:34,348][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:11:34,846][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:11:35,346][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:11:35,847][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10334 tokens.
[2025-11-13 02:11:36,577][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.08%, ΔTime: 00:00:32
[2025-11-13 02:11:37,357][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:11:37,359][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:11:37,360][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:11:38,344][__main__][INFO] - Iteration 272 took 54s (33.77% Gen, 64.42% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 15m 57s. Estimated total time: 45h 17m 54s. Time estimates for 10 more iterations: 9m 3s, 100 more iterations: 1h 30m 35s, 500 more iterations: 7h 32m 59s.
[2025-11-13 02:11:38,347][__main__][INFO] - Starting iteration 272.
[2025-11-13 02:11:38,851][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:11:38,851][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:11:57,412][__main__][INFO] - Number of regex retries in iteration 272: 0
[2025-11-13 02:11:57,412][__main__][INFO] - agents played in iteration 272 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:11:58,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:11:58,227][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:11:58,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:11:58,274][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:11:58,275][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:11:58,276][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:11:58,935][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:11:59,390][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:11:59,905][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:12:00,409][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:12:00,909][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:12:01,411][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:12:01,916][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:12:02,417][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:12:02,913][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:12:03,415][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:12:03,925][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:12:04,425][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:12:04,928][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:12:05,428][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:12:05,926][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:12:06,427][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:12:06,928][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:12:07,429][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:12:07,927][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:12:08,427][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:12:08,927][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:12:09,424][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:12:09,923][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:12:10,424][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:12:10,925][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:12:11,424][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:12:11,919][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:12:12,416][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:12:12,920][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:12:13,417][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:12:13,917][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:12:14,416][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:12:14,913][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:12:15,411][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:12:15,914][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:12:16,415][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:12:16,929][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:12:17,433][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:12:17,932][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:12:18,437][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:12:18,942][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:12:19,444][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:12:19,945][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:12:20,447][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:12:20,954][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:12:21,454][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:12:21,956][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:12:22,455][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:12:22,957][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:12:23,454][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:12:23,954][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:12:24,451][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:12:24,953][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:12:25,451][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:12:25,949][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:12:26,447][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:12:26,945][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:12:27,446][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:12:27,947][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:12:28,447][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:12:28,944][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:12:29,444][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:12:29,943][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:12:30,444][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:12:30,942][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10246 tokens.
[2025-11-13 02:12:31,676][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 02:12:32,460][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:12:32,462][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:12:32,464][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:12:33,380][__main__][INFO] - Iteration 273 took 54s (34.04% Gen, 64.28% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 23m 39s. Estimated total time: 45h 26m 30s. Time estimates for 10 more iterations: 9m 5s, 100 more iterations: 1h 30m 53s, 500 more iterations: 7h 34m 25s.
[2025-11-13 02:12:33,383][__main__][INFO] - Starting iteration 273.
[2025-11-13 02:12:33,876][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:12:33,877][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:12:50,764][__main__][INFO] - Number of regex retries in iteration 273: 0
[2025-11-13 02:12:50,764][__main__][INFO] - agents played in iteration 273 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:12:51,642][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:12:51,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:12:51,695][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:12:51,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:12:51,719][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:12:51,719][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:12:52,394][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:12:52,849][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:12:53,352][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:12:53,855][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:12:54,356][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:12:54,855][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:12:55,355][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:12:55,852][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:12:56,352][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:12:56,856][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:12:57,357][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:12:57,860][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:12:58,361][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:12:58,863][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:12:59,365][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:12:59,863][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:13:00,367][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:13:00,867][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:13:01,363][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:13:01,859][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:13:02,356][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:13:02,858][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:13:03,355][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:13:03,851][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:13:04,349][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:13:04,844][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:13:05,343][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:13:05,839][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:13:06,336][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:13:06,837][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:13:07,332][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:13:07,829][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:13:08,326][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:13:08,823][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:13:09,334][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:13:09,831][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:13:10,327][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:13:10,824][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:13:11,324][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:13:11,832][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:13:12,332][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:13:12,832][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:13:13,346][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:13:13,848][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:13:14,348][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:13:14,847][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:13:15,345][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:13:15,848][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:13:16,347][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:13:16,846][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:13:17,352][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:13:17,853][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:13:18,350][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:13:18,850][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:13:19,347][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:13:19,844][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:13:20,340][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:13:20,837][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:13:21,339][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:13:21,837][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:13:22,338][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:13:22,837][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:13:23,335][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:13:23,835][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:13:24,332][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10191 tokens.
[2025-11-13 02:13:25,078][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 61.99%, ΔTime: 00:00:32
[2025-11-13 02:13:25,849][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:13:25,851][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:13:25,852][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:13:27,042][__main__][INFO] - Iteration 274 took 53s (31.76% Gen, 66.00% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 14m 33s. Estimated total time: 44h 18m 19s. Time estimates for 10 more iterations: 8m 51s, 100 more iterations: 1h 28m 36s, 500 more iterations: 7h 23m 3s.
[2025-11-13 02:13:27,044][__main__][INFO] - Starting iteration 274.
[2025-11-13 02:13:27,517][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:13:27,518][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:13:31,480][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:13:44,387][__main__][INFO] - Number of regex retries in iteration 274: 1
[2025-11-13 02:13:44,388][__main__][INFO] - agents played in iteration 274 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:13:45,249][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:13:45,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:13:45,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:13:45,325][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:13:45,325][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:13:45,326][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:13:45,992][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:13:46,450][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:13:46,952][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:13:47,459][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:13:47,956][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:13:48,456][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:13:48,956][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:13:49,454][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:13:49,952][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:13:50,464][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:13:50,960][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:13:51,458][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:13:51,961][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:13:52,457][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:13:52,964][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:13:53,460][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:13:53,958][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:13:54,466][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:13:54,966][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:13:55,467][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:13:55,967][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:13:56,465][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:13:56,970][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:13:57,468][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:13:57,968][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:13:58,466][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:13:58,964][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:13:59,465][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:13:59,960][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:14:00,460][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:14:00,967][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:14:01,467][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:14:01,972][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:14:02,476][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:14:02,978][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:14:03,482][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:14:03,982][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:14:04,484][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:14:04,990][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:14:05,492][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:14:06,000][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:14:06,502][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:14:07,003][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:14:07,507][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:14:08,007][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:14:08,506][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:14:09,005][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:14:09,505][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:14:10,007][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:14:10,511][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:14:11,012][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:14:11,517][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:14:12,013][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:14:12,511][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:14:13,014][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:14:13,512][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:14:14,009][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:14:14,507][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:14:15,006][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:14:15,505][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:14:16,006][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:14:16,506][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:14:17,005][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:14:17,502][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:14:18,002][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10316 tokens.
[2025-11-13 02:14:18,771][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 02:14:19,542][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:14:19,544][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:14:19,545][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:14:20,482][__main__][INFO] - Iteration 275 took 52s (31.85% Gen, 66.38% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 3m 37s. Estimated total time: 44h 8m 16s. Time estimates for 10 more iterations: 8m 49s, 100 more iterations: 1h 28m 16s, 500 more iterations: 7h 21m 22s.
[2025-11-13 02:14:20,485][__main__][INFO] - Starting iteration 275.
[2025-11-13 02:14:20,962][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:14:20,963][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:14:38,715][__main__][INFO] - Number of regex retries in iteration 275: 0
[2025-11-13 02:14:38,716][__main__][INFO] - agents played in iteration 275 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:14:39,569][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:14:39,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:14:39,619][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:14:39,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:14:39,642][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:14:39,642][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:14:40,304][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:14:40,760][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:14:41,264][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:14:41,766][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:14:42,267][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:14:42,764][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:14:43,275][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:14:43,772][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:14:44,274][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:14:44,781][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:14:45,282][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:14:45,781][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:14:46,280][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:14:46,781][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:14:47,285][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:14:47,788][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:14:48,301][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:14:48,810][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:14:49,308][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:14:49,809][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:14:50,308][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:14:50,807][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:14:51,307][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:14:51,809][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:14:52,309][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:14:52,814][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:14:53,318][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:14:53,819][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:14:54,320][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:14:54,822][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:14:55,326][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:14:55,825][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:14:56,326][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:14:56,827][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:14:57,331][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:14:57,846][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:14:58,344][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:14:58,845][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:14:59,351][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:14:59,853][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:15:00,365][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:15:00,862][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:15:01,363][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:15:01,879][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:15:02,380][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:15:02,883][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:15:03,384][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:15:03,882][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:15:04,387][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:15:04,890][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:15:05,390][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:15:05,890][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:15:06,388][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:15:06,891][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:15:07,389][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:15:07,886][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:15:08,388][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:15:08,886][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:15:09,386][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:15:09,885][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:15:10,383][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:15:10,884][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:15:11,381][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:15:11,878][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:15:12,380][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10272 tokens.
[2025-11-13 02:15:13,075][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 02:15:13,910][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:15:13,912][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:15:13,919][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:15:15,002][__main__][INFO] - Iteration 276 took 54s (32.85% Gen, 65.14% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 56m 28s. Estimated total time: 45h 2m 1s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 4s, 500 more iterations: 7h 30m 20s.
[2025-11-13 02:15:15,004][__main__][INFO] - Starting iteration 276.
[2025-11-13 02:15:15,497][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:15:15,497][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:15:34,004][__main__][INFO] - Number of regex retries in iteration 276: 0
[2025-11-13 02:15:34,004][__main__][INFO] - agents played in iteration 276 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:15:34,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:15:34,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:15:34,937][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:15:34,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:15:34,960][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:15:34,961][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:15:35,642][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:15:36,099][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:15:36,603][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:15:37,107][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:15:37,607][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:15:38,114][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:15:38,618][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:15:39,118][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:15:39,618][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:15:40,117][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:15:40,618][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:15:41,118][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:15:41,617][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:15:42,116][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:15:42,615][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:15:43,116][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:15:43,619][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:15:44,119][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:15:44,617][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:15:45,121][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:15:45,621][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:15:46,120][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:15:46,621][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:15:47,119][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:15:47,623][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:15:48,123][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:15:48,623][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:15:49,123][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:15:49,621][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:15:50,119][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:15:50,616][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:15:51,114][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:15:51,612][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:15:52,109][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:15:52,606][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:15:53,109][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:15:53,610][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:15:54,109][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:15:54,613][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:15:55,118][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:15:55,621][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:15:56,124][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:15:56,626][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:15:57,127][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:15:57,627][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:15:58,127][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:15:58,631][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:15:59,129][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:15:59,628][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:16:00,136][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:16:00,639][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:16:01,144][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:16:01,649][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:16:02,153][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:16:02,653][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:16:03,149][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:16:03,657][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:16:04,159][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:16:04,656][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:16:05,158][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:16:05,654][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:16:06,154][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:16:06,654][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:16:07,149][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:16:07,647][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10280 tokens.
[2025-11-13 02:16:08,353][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 02:16:09,149][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:16:09,151][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:16:09,153][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:16:10,154][__main__][INFO] - Iteration 277 took 54s (33.86% Gen, 64.31% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 26m 25s. Estimated total time: 45h 32m 54s. Time estimates for 10 more iterations: 9m 6s, 100 more iterations: 1h 31m 5s, 500 more iterations: 7h 35m 29s.
[2025-11-13 02:16:10,156][__main__][INFO] - Starting iteration 277.
[2025-11-13 02:16:10,634][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:16:10,634][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:16:28,148][__main__][INFO] - Number of regex retries in iteration 277: 0
[2025-11-13 02:16:28,149][__main__][INFO] - agents played in iteration 277 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:16:28,925][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:16:28,953][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:16:28,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:16:29,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:16:29,003][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:16:29,003][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:16:29,673][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:16:30,123][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:16:30,625][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:16:31,122][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:16:31,620][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:16:32,120][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:16:32,618][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:16:33,118][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:16:33,622][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:16:34,128][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:16:34,631][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:16:35,132][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:16:35,637][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:16:36,141][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:16:36,642][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:16:37,143][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:16:37,648][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:16:38,149][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:16:38,656][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:16:39,160][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:16:39,661][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:16:40,164][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:16:40,666][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:16:41,183][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:16:41,691][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:16:42,191][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:16:42,705][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:16:43,205][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:16:43,705][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:16:44,209][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:16:44,711][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:16:45,215][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:16:45,720][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:16:46,221][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:16:46,721][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:16:47,220][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:16:47,722][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:16:48,226][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:16:48,729][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:16:49,236][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:16:49,741][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:16:50,246][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:16:50,749][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:16:51,251][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:16:51,751][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:16:52,255][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:16:52,759][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:16:53,260][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:16:53,762][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:16:54,259][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:16:54,763][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:16:55,263][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:16:55,760][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:16:56,267][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:16:56,767][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:16:57,264][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:16:57,760][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:16:58,259][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:16:58,762][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:16:59,263][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:16:59,763][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:17:00,260][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:17:00,759][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:17:01,259][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:17:01,760][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10321 tokens.
[2025-11-13 02:17:02,466][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 02:17:03,224][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:17:03,225][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:17:03,227][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:17:04,239][__main__][INFO] - Iteration 278 took 53s (32.67% Gen, 65.44% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 32m 56s. Estimated total time: 44h 40m 19s. Time estimates for 10 more iterations: 8m 56s, 100 more iterations: 1h 29m 20s, 500 more iterations: 7h 26m 43s.
[2025-11-13 02:17:04,241][__main__][INFO] - Starting iteration 278.
[2025-11-13 02:17:04,721][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:17:04,721][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:17:20,030][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:17:22,773][__main__][INFO] - Number of regex retries in iteration 278: 1
[2025-11-13 02:17:22,774][__main__][INFO] - agents played in iteration 278 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:17:23,551][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:17:23,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:17:23,595][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:17:23,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:17:23,617][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:17:23,618][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:17:24,269][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:17:24,729][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:17:25,238][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:17:25,736][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:17:26,242][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:17:26,746][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:17:27,247][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:17:27,752][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:17:28,256][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:17:28,759][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:17:29,259][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:17:29,801][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:17:30,304][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:17:30,804][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:17:31,309][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:17:31,809][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:17:32,312][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:17:32,810][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:17:33,308][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:17:33,817][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:17:34,321][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:17:34,824][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:17:35,322][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:17:35,819][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:17:36,320][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:17:36,820][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:17:37,319][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:17:37,825][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:17:38,324][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:17:38,830][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:17:39,330][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:17:39,830][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:17:40,332][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:17:40,833][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:17:41,332][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:17:41,832][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:17:42,335][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:17:42,833][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:17:43,332][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:17:43,832][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:17:44,347][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:17:44,853][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:17:45,359][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:17:45,862][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:17:46,365][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:17:46,867][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:17:47,369][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:17:47,869][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:17:48,372][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:17:48,873][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:17:49,373][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:17:49,886][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:17:50,384][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:17:50,897][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:17:51,396][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:17:51,894][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:17:52,395][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:17:52,894][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:17:53,394][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:17:53,894][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:17:54,389][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:17:54,904][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:17:55,401][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:17:55,901][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:17:56,400][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10281 tokens.
[2025-11-13 02:17:57,094][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 02:17:57,875][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:17:57,876][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:17:57,878][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:17:58,815][__main__][INFO] - Iteration 279 took 54s (33.37% Gen, 64.89% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 56m 27s. Estimated total time: 45h 4m 44s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 9s, 500 more iterations: 7h 30m 47s.
[2025-11-13 02:17:58,817][__main__][INFO] - Starting iteration 279.
[2025-11-13 02:17:59,301][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:17:59,301][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:18:04,428][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:18:17,590][__main__][INFO] - Number of regex retries in iteration 279: 1
[2025-11-13 02:18:17,591][__main__][INFO] - agents played in iteration 279 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:18:18,475][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:18:18,498][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:18:18,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:18:18,542][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:18:18,542][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:18:18,543][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:18:19,244][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:18:19,706][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:18:20,214][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:18:20,711][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:18:21,225][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:18:21,732][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:18:22,232][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:18:22,732][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:18:23,235][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:18:23,739][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:18:24,240][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:18:24,740][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:18:25,241][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:18:25,741][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:18:26,240][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:18:26,743][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:18:27,242][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:18:27,749][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:18:28,249][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:18:28,749][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:18:29,257][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:18:29,760][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:18:30,261][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:18:30,763][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:18:31,266][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:18:31,770][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:18:32,270][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:18:32,774][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:18:33,279][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:18:33,782][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:18:34,285][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:18:34,787][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:18:35,288][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:18:35,799][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:18:36,302][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:18:36,816][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:18:37,314][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:18:37,816][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:18:38,324][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:18:38,820][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:18:39,320][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:18:39,818][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:18:40,319][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:18:40,830][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:18:41,330][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:18:41,830][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:18:42,334][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:18:42,833][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:18:43,338][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:18:43,840][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:18:44,340][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:18:44,849][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:18:45,350][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:18:45,851][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:18:46,351][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:18:46,851][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:18:47,353][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:18:47,851][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:18:48,351][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:18:48,853][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:18:49,348][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:18:49,848][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:18:50,347][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:18:50,849][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:18:51,348][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10226 tokens.
[2025-11-13 02:18:52,053][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 02:18:52,829][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:18:52,831][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:18:52,832][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:18:53,752][__main__][INFO] - Iteration 280 took 54s (33.59% Gen, 64.72% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 13m 23s. Estimated total time: 45h 22m 35s. Time estimates for 10 more iterations: 9m 4s, 100 more iterations: 1h 30m 45s, 500 more iterations: 7h 33m 45s.
[2025-11-13 02:18:53,754][__main__][INFO] - Starting iteration 280.
[2025-11-13 02:18:54,246][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 02:18:54,247][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:19:11,574][__main__][INFO] - Number of regex retries in iteration 280: 0
[2025-11-13 02:19:11,575][__main__][INFO] - agents played in iteration 280 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:19:12,427][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:19:12,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:19:12,479][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:19:12,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:19:12,502][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:19:12,503][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:19:13,218][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:19:13,679][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:19:14,184][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:19:14,685][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:19:15,186][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:19:15,688][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:19:16,187][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:19:16,686][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:19:17,191][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:19:17,700][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:19:18,203][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:19:18,703][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:19:19,203][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:19:19,703][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:19:20,230][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:19:20,730][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:19:21,232][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:19:21,740][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:19:22,246][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:19:22,745][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:19:23,244][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:19:23,746][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:19:24,247][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:19:24,745][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:19:25,242][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:19:25,743][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:19:26,242][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:19:26,781][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:19:27,282][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:19:27,782][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:19:28,294][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:19:28,795][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:19:29,300][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:19:29,800][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:19:30,299][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:19:30,801][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:19:31,300][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:19:31,798][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:19:32,305][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:19:32,805][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:19:33,302][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:19:33,802][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:19:34,296][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:19:34,795][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:19:35,292][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:19:35,792][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:19:36,293][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:19:36,792][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:19:37,294][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:19:37,792][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:19:38,289][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:19:38,792][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:19:39,293][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:19:39,794][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:19:40,293][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:19:40,791][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:19:41,290][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:19:41,789][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:19:42,289][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:19:42,788][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:19:43,284][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:19:43,784][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:19:44,282][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:19:44,780][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:19:45,278][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10263 tokens.
[2025-11-13 02:19:45,977][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 02:19:46,751][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:19:46,753][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:19:46,755][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:19:48,534][__main__][INFO] - Iteration 281 took 54s (31.92% Gen, 64.80% Train). Generation: 17s, Training: 35s. Estimated remaining time: 41h 4m 18s. Estimated total time: 45h 14m 25s. Time estimates for 10 more iterations: 9m 2s, 100 more iterations: 1h 30m 28s, 500 more iterations: 7h 32m 24s.
[2025-11-13 02:19:48,536][__main__][INFO] - Starting iteration 281.
[2025-11-13 02:19:49,023][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:19:49,024][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:19:53,448][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:20:05,550][__main__][INFO] - Number of regex retries in iteration 281: 1
[2025-11-13 02:20:05,551][__main__][INFO] - agents played in iteration 281 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:20:06,345][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:20:06,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:20:06,391][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:20:06,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:20:06,413][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:20:06,414][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:20:07,116][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:20:07,581][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:20:08,086][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:20:08,583][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:20:09,081][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:20:09,579][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:20:10,081][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:20:10,583][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:20:11,083][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:20:11,591][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:20:12,099][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:20:12,601][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:20:13,107][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:20:13,609][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:20:14,110][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:20:14,613][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:20:15,115][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:20:15,627][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:20:16,127][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:20:16,626][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:20:17,132][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:20:17,630][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:20:18,143][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:20:18,641][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:20:19,141][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:20:19,667][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:20:20,168][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:20:20,671][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:20:21,171][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:20:21,670][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:20:22,172][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:20:22,671][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:20:23,170][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:20:23,670][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:20:24,165][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:20:24,666][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:20:25,165][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:20:25,667][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:20:26,163][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:20:26,661][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:20:27,163][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:20:27,662][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:20:28,158][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:20:28,664][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:20:29,163][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:20:29,662][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:20:30,158][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:20:30,658][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:20:31,164][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:20:31,664][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:20:32,166][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:20:32,670][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:20:33,174][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:20:33,678][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:20:34,180][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:20:34,680][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:20:35,184][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:20:35,684][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:20:36,184][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:20:36,687][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:20:37,185][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:20:37,685][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:20:38,186][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:20:38,686][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:20:39,188][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10450 tokens.
[2025-11-13 02:20:39,876][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 02:20:40,671][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:20:40,672][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:20:40,674][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:20:41,888][__main__][INFO] - Iteration 282 took 52s (31.26% Gen, 66.44% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 52m 17s. Estimated total time: 44h 3m 18s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 6s, 500 more iterations: 7h 20m 33s.
[2025-11-13 02:20:41,890][__main__][INFO] - Starting iteration 282.
[2025-11-13 02:20:42,368][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:20:42,369][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:21:01,419][__main__][INFO] - Number of regex retries in iteration 282: 0
[2025-11-13 02:21:01,420][__main__][INFO] - agents played in iteration 282 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:21:02,211][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:21:02,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:21:02,265][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:21:02,288][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:21:02,289][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:21:02,290][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:21:02,989][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:21:03,439][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:21:03,951][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:21:04,448][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:21:04,950][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:21:05,464][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:21:05,966][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:21:06,472][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:21:06,970][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:21:07,467][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:21:07,977][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:21:08,477][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:21:08,976][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:21:09,477][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:21:09,977][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:21:10,481][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:21:10,983][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:21:11,484][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:21:11,986][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:21:12,484][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:21:12,981][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:21:13,476][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:21:13,977][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:21:14,479][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:21:14,977][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:21:15,496][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:21:15,998][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:21:16,500][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:21:17,000][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:21:17,497][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:21:17,997][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:21:18,494][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:21:18,993][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:21:19,493][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:21:19,992][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:21:20,490][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:21:20,989][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:21:21,490][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:21:21,988][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:21:22,489][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:21:22,986][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:21:23,485][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:21:23,982][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:21:24,486][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:21:24,986][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:21:25,486][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:21:25,985][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:21:26,487][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:21:26,992][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:21:27,503][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:21:28,018][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:21:28,533][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:21:29,046][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:21:29,560][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:21:30,074][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:21:30,574][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:21:31,078][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:21:31,581][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:21:32,082][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:21:32,581][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:21:33,082][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:21:33,582][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:21:34,080][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:21:34,578][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:21:35,078][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10412 tokens.
[2025-11-13 02:21:35,778][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 02:21:36,547][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:21:36,548][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:21:36,550][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:21:37,522][__main__][INFO] - Iteration 283 took 55s (34.54% Gen, 63.69% Train). Generation: 19s, Training: 35s. Estimated remaining time: 41h 45m 46s. Estimated total time: 45h 57m 42s. Time estimates for 10 more iterations: 9m 11s, 100 more iterations: 1h 31m 55s, 500 more iterations: 7h 39m 37s.
[2025-11-13 02:21:37,524][__main__][INFO] - Starting iteration 283.
[2025-11-13 02:21:38,034][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:21:38,035][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:21:45,096][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:21:55,651][__main__][INFO] - Number of regex retries in iteration 283: 1
[2025-11-13 02:21:55,651][__main__][INFO] - agents played in iteration 283 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:21:56,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:21:56,540][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:21:56,566][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:21:56,588][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:21:56,589][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:21:56,590][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:21:57,267][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:21:57,725][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:21:58,233][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:21:58,736][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:21:59,235][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:21:59,735][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:22:00,239][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:22:00,738][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:22:01,241][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:22:01,756][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:22:02,258][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:22:02,770][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:22:03,271][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:22:03,773][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:22:04,284][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:22:04,786][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:22:05,285][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:22:05,785][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:22:06,284][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:22:06,781][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:22:07,278][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:22:07,775][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:22:08,273][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:22:08,769][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:22:09,273][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:22:09,768][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:22:10,264][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:22:10,780][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:22:11,278][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:22:11,776][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:22:12,276][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:22:12,776][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:22:13,282][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:22:13,778][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:22:14,278][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:22:14,786][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:22:15,284][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:22:15,783][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:22:16,281][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:22:16,778][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:22:17,290][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:22:17,789][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:22:18,288][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:22:18,788][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:22:19,289][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:22:19,794][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:22:20,293][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:22:20,793][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:22:21,298][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:22:21,801][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:22:22,305][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:22:22,803][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:22:23,304][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:22:23,807][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:22:24,307][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:22:24,810][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:22:25,313][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:22:25,812][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:22:26,312][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:22:26,810][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:22:27,311][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:22:27,814][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:22:28,317][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:22:28,817][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:22:29,323][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10323 tokens.
[2025-11-13 02:22:30,040][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 02:22:30,832][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:22:30,834][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:22:30,836][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:22:31,765][__main__][INFO] - Iteration 284 took 53s (32.78% Gen, 65.48% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 33m 44s. Estimated total time: 44h 46m 34s. Time estimates for 10 more iterations: 8m 57s, 100 more iterations: 1h 29m 33s, 500 more iterations: 7h 27m 45s.
[2025-11-13 02:22:31,767][__main__][INFO] - Starting iteration 284.
[2025-11-13 02:22:32,236][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:22:32,237][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:22:49,439][__main__][INFO] - Number of regex retries in iteration 284: 0
[2025-11-13 02:22:49,439][__main__][INFO] - agents played in iteration 284 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:22:50,266][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:22:50,291][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:22:50,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:22:50,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:22:50,338][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:22:50,338][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:22:51,014][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:22:51,465][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:22:51,968][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:22:52,473][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:22:52,972][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:22:53,473][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:22:53,988][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:22:54,493][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:22:55,000][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:22:55,501][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:22:56,000][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:22:56,507][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:22:57,008][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:22:57,513][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:22:58,017][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:22:58,517][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:22:59,023][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:22:59,528][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:23:00,026][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:23:00,527][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:23:01,028][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:23:01,525][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:23:02,025][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:23:02,521][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:23:03,020][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:23:03,515][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:23:04,010][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:23:04,511][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:23:05,007][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:23:05,507][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:23:06,003][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:23:06,506][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:23:07,004][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:23:07,505][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:23:08,006][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:23:08,506][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:23:09,011][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:23:09,510][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:23:10,009][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:23:10,508][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:23:11,011][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:23:11,507][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:23:12,004][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:23:12,508][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:23:13,004][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:23:13,507][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:23:14,007][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:23:14,503][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:23:15,000][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:23:15,502][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:23:16,005][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:23:16,504][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:23:17,003][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:23:17,507][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:23:18,006][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:23:18,505][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:23:19,006][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:23:19,505][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:23:20,007][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:23:20,511][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:23:21,012][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:23:21,510][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:23:22,008][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:23:22,510][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:23:23,015][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10361 tokens.
[2025-11-13 02:23:23,751][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 02:23:24,510][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:23:24,511][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:23:24,513][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:23:25,718][__main__][INFO] - Iteration 285 took 53s (32.16% Gen, 65.58% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 20m 23s. Estimated total time: 44h 34m 7s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 8s, 500 more iterations: 7h 25m 41s.
[2025-11-13 02:23:25,720][__main__][INFO] - Starting iteration 285.
[2025-11-13 02:23:26,208][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:23:26,209][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:23:44,449][__main__][INFO] - Number of regex retries in iteration 285: 0
[2025-11-13 02:23:44,450][__main__][INFO] - agents played in iteration 285 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:23:45,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:23:45,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:23:45,297][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:23:45,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:23:45,320][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:23:45,321][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:23:46,003][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:23:46,460][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:23:46,963][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:23:47,464][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:23:47,970][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:23:48,477][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:23:48,982][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:23:49,490][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:23:49,987][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:23:50,490][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:23:50,997][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:23:51,498][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:23:52,017][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:23:52,547][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:23:53,051][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:23:53,555][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:23:54,062][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:23:54,567][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:23:55,064][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:23:55,564][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:23:56,064][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:23:56,561][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:23:57,061][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:23:57,561][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:23:58,056][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:23:58,560][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:23:59,055][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:23:59,555][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:24:00,051][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:24:00,550][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:24:01,049][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:24:01,544][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:24:02,040][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:24:02,543][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:24:03,042][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:24:03,539][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:24:04,036][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:24:04,536][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:24:05,037][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:24:05,536][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:24:06,035][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:24:06,535][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:24:07,031][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:24:07,534][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:24:08,030][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:24:08,529][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:24:09,027][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:24:09,523][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:24:10,027][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:24:10,527][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:24:11,022][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:24:11,528][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:24:12,027][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:24:12,527][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:24:13,029][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:24:13,528][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:24:14,032][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:24:14,532][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:24:15,031][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:24:15,528][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:24:16,024][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:24:16,529][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:24:17,026][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:24:17,527][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:24:18,030][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10388 tokens.
[2025-11-13 02:24:18,750][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 02:24:19,519][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:24:19,522][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:24:19,524][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:24:20,438][__main__][INFO] - Iteration 286 took 54s (33.64% Gen, 64.68% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 56m 54s. Estimated total time: 45h 11m 32s. Time estimates for 10 more iterations: 9m 2s, 100 more iterations: 1h 30m 23s, 500 more iterations: 7h 31m 55s.
[2025-11-13 02:24:20,441][__main__][INFO] - Starting iteration 286.
[2025-11-13 02:24:20,908][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:24:20,909][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:24:40,470][__main__][INFO] - Number of regex retries in iteration 286: 0
[2025-11-13 02:24:40,471][__main__][INFO] - agents played in iteration 286 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:24:41,322][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:24:41,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:24:41,374][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:24:41,397][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:24:41,398][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:24:41,399][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:24:42,074][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:24:42,530][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:24:43,033][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:24:43,529][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:24:44,035][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:24:44,539][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:24:45,040][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:24:45,540][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:24:46,039][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:24:46,546][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:24:47,044][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:24:47,543][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:24:48,049][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:24:48,546][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:24:49,052][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:24:49,550][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:24:50,052][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:24:50,553][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:24:51,054][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:24:51,556][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:24:52,059][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:24:52,561][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:24:53,063][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:24:53,565][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:24:54,067][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:24:54,566][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:24:55,069][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:24:55,577][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:24:56,077][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:24:56,577][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:24:57,080][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:24:57,580][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:24:58,080][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:24:58,579][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:24:59,077][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:24:59,576][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:25:00,074][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:25:00,568][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:25:01,066][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:25:01,562][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:25:02,061][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:25:02,557][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:25:03,054][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:25:03,554][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:25:04,052][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:25:04,547][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:25:05,044][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:25:05,540][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:25:06,045][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:25:06,544][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:25:07,045][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:25:07,550][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:25:08,051][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:25:08,557][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:25:09,061][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:25:09,561][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:25:10,066][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:25:10,596][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:25:11,096][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:25:11,611][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:25:12,114][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:25:12,634][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:25:13,138][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:25:13,643][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:25:14,146][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10460 tokens.
[2025-11-13 02:25:14,883][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.37%, ΔTime: 00:00:32
[2025-11-13 02:25:15,674][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:25:15,676][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:25:15,679][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:25:16,572][__main__][INFO] - Iteration 287 took 55s (35.14% Gen, 63.25% Train). Generation: 19s, Training: 35s. Estimated remaining time: 42h 7m 38s. Estimated total time: 46h 23m 13s. Time estimates for 10 more iterations: 9m 16s, 100 more iterations: 1h 32m 46s, 500 more iterations: 7h 43m 52s.
[2025-11-13 02:25:16,574][__main__][INFO] - Starting iteration 287.
[2025-11-13 02:25:17,041][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:25:17,042][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:25:21,049][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:25:23,644][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:25:32,904][__main__][INFO] - Number of regex retries in iteration 287: 2
[2025-11-13 02:25:32,905][__main__][INFO] - agents played in iteration 287 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:25:33,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:25:33,762][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:25:33,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:25:33,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:25:33,806][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:25:33,807][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:25:34,470][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:25:34,937][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:25:35,437][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:25:35,932][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:25:36,433][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:25:36,931][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:25:37,437][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:25:37,934][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:25:38,440][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:25:38,941][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:25:39,440][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:25:39,937][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:25:40,440][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:25:40,938][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:25:41,451][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:25:41,952][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:25:42,450][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:25:42,958][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:25:43,461][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:25:43,963][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:25:44,465][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:25:44,964][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:25:45,466][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:25:45,965][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:25:46,466][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:25:46,966][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:25:47,464][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:25:47,967][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:25:48,468][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:25:48,964][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:25:49,462][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:25:49,964][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:25:50,459][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:25:50,958][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:25:51,462][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:25:51,960][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:25:52,458][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:25:52,956][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:25:53,461][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:25:53,957][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:25:54,457][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:25:54,959][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:25:55,456][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:25:55,953][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:25:56,451][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:25:56,947][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:25:57,451][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:25:57,948][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:25:58,448][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:25:58,944][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:25:59,443][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:25:59,947][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:26:00,442][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:26:00,942][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:26:01,445][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:26:01,941][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:26:02,442][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:26:02,957][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:26:03,460][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:26:03,963][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:26:04,468][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:26:04,973][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:26:05,473][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:26:05,980][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:26:06,483][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10314 tokens.
[2025-11-13 02:26:07,278][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-13 02:26:08,032][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:26:08,034][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:26:08,035][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:26:08,885][__main__][INFO] - Iteration 288 took 51s (30.60% Gen, 67.76% Train). Generation: 15s, Training: 35s. Estimated remaining time: 38h 55m 46s. Estimated total time: 43h 12m 13s. Time estimates for 10 more iterations: 8m 38s, 100 more iterations: 1h 26m 24s, 500 more iterations: 7h 12m 2s.
[2025-11-13 02:26:08,887][__main__][INFO] - Starting iteration 288.
[2025-11-13 02:26:09,367][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:26:09,368][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:26:26,297][__main__][INFO] - Number of regex retries in iteration 288: 0
[2025-11-13 02:26:26,297][__main__][INFO] - agents played in iteration 288 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:26:27,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:26:27,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:26:27,192][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:26:27,215][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:26:27,216][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:26:27,217][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:26:27,900][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:26:28,358][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:26:28,863][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:26:29,364][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:26:29,864][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:26:30,367][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:26:30,867][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:26:31,366][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:26:31,868][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:26:32,363][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:26:32,862][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:26:33,355][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:26:33,852][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:26:34,355][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:26:34,851][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:26:35,348][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:26:35,861][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:26:36,364][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:26:36,865][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:26:37,370][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:26:37,876][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:26:38,373][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:26:38,873][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:26:39,374][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:26:39,879][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:26:40,383][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:26:40,890][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:26:41,394][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:26:41,898][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:26:42,399][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:26:42,902][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:26:43,406][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:26:43,912][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:26:44,413][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:26:44,916][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:26:45,417][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:26:45,920][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:26:46,432][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:26:46,930][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:26:47,429][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:26:47,936][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:26:48,442][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:26:48,952][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:26:49,456][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:26:49,957][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:26:50,465][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:26:50,966][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:26:51,469][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:26:51,968][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:26:52,468][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:26:52,970][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:26:53,467][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:26:53,968][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:26:54,465][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:26:54,965][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:26:55,464][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:26:55,966][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:26:56,470][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:26:56,973][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:26:57,475][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:26:57,980][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:26:58,483][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:26:58,985][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:26:59,495][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:26:59,997][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10377 tokens.
[2025-11-13 02:27:00,782][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 02:27:01,526][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:27:01,528][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:27:01,529][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:27:02,443][__main__][INFO] - Iteration 289 took 53s (31.90% Gen, 66.38% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 56m 29s. Estimated total time: 44h 13m 50s. Time estimates for 10 more iterations: 8m 50s, 100 more iterations: 1h 28m 27s, 500 more iterations: 7h 22m 18s.
[2025-11-13 02:27:02,445][__main__][INFO] - Starting iteration 289.
[2025-11-13 02:27:02,918][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:27:02,919][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:27:13,902][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:27:20,219][__main__][INFO] - Number of regex retries in iteration 289: 1
[2025-11-13 02:27:20,220][__main__][INFO] - agents played in iteration 289 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:27:21,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:27:21,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:27:21,057][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:27:21,080][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:27:21,081][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:27:21,082][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:27:21,768][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:27:22,229][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:27:22,731][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:27:23,232][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:27:23,733][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:27:24,234][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:27:24,733][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:27:25,232][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:27:25,745][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:27:26,245][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:27:26,742][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:27:27,245][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:27:27,745][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:27:28,254][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:27:28,755][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:27:29,251][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:27:29,752][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:27:30,248][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:27:30,744][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:27:31,247][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:27:31,747][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:27:32,257][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:27:32,758][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:27:33,260][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:27:33,759][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:27:34,261][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:27:34,759][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:27:35,257][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:27:35,756][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:27:36,268][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:27:36,769][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:27:37,272][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:27:37,776][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:27:38,274][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:27:38,771][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:27:39,267][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:27:39,764][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:27:40,266][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:27:40,764][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:27:41,258][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:27:41,759][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:27:42,259][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:27:42,759][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:27:43,258][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:27:43,754][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:27:44,265][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:27:44,763][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:27:45,263][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:27:45,764][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:27:46,263][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:27:46,769][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:27:47,269][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:27:47,770][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:27:48,270][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:27:48,768][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:27:49,271][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:27:49,768][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:27:50,269][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:27:50,772][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:27:51,275][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:27:51,777][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:27:52,284][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:27:52,784][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:27:53,289][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:27:53,792][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10372 tokens.
[2025-11-13 02:27:54,594][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 02:27:55,337][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:27:55,339][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:27:55,341][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:27:56,206][__main__][INFO] - Iteration 290 took 53s (32.46% Gen, 65.91% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 6m 11s. Estimated total time: 44h 24m 26s. Time estimates for 10 more iterations: 8m 52s, 100 more iterations: 1h 28m 48s, 500 more iterations: 7h 24m 4s.
[2025-11-13 02:27:56,208][__main__][INFO] - Starting iteration 290.
[2025-11-13 02:27:56,690][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 02:27:56,691][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:28:13,598][__main__][INFO] - Number of regex retries in iteration 290: 0
[2025-11-13 02:28:13,599][__main__][INFO] - agents played in iteration 290 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:28:14,409][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:28:14,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:28:14,462][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:28:14,485][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:28:14,486][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:28:14,487][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:28:15,156][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:28:15,610][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:28:16,113][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:28:16,611][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:28:17,113][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:28:17,609][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:28:18,108][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:28:18,606][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:28:19,104][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:28:19,603][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:28:20,101][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:28:20,600][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:28:21,100][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:28:21,596][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:28:22,098][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:28:22,593][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:28:23,092][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:28:23,595][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:28:24,093][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:28:24,593][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:28:25,092][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:28:25,590][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:28:26,093][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:28:26,592][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:28:27,095][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:28:27,593][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:28:28,094][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:28:28,592][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:28:29,094][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:28:29,590][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:28:30,091][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:28:30,591][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:28:31,090][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:28:31,597][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:28:32,101][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:28:32,605][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:28:33,109][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:28:33,609][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:28:34,110][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:28:34,610][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:28:35,114][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:28:35,616][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:28:36,119][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:28:36,622][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:28:37,123][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:28:37,626][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:28:38,151][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:28:38,653][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:28:39,169][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:28:39,674][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:28:40,177][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:28:40,677][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:28:41,177][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:28:41,687][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:28:42,189][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:28:42,692][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:28:43,199][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:28:43,701][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:28:44,207][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:28:44,715][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:28:45,217][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:28:45,722][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:28:46,224][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:28:46,727][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:28:47,231][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10388 tokens.
[2025-11-13 02:28:48,058][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.44%, ΔTime: 00:00:32
[2025-11-13 02:28:48,815][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:28:48,817][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:28:48,819][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:28:50,576][__main__][INFO] - Iteration 291 took 53s (31.38% Gen, 65.36% Train). Generation: 16s, Training: 35s. Estimated remaining time: 40h 35m 10s. Estimated total time: 44h 54m 19s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 48s, 500 more iterations: 7h 29m 3s.
[2025-11-13 02:28:50,578][__main__][INFO] - Starting iteration 291.
[2025-11-13 02:28:51,074][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:28:51,074][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:29:08,885][__main__][INFO] - Number of regex retries in iteration 291: 0
[2025-11-13 02:29:08,886][__main__][INFO] - agents played in iteration 291 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:29:09,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:29:09,695][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:29:09,717][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:29:09,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:29:09,740][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:29:09,741][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:29:10,468][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:29:10,929][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:29:11,438][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:29:11,938][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:29:12,441][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:29:12,942][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:29:13,447][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:29:13,946][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:29:14,447][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:29:14,947][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:29:15,447][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:29:15,960][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:29:16,458][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:29:16,957][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:29:17,467][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:29:17,967][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:29:18,479][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:29:18,975][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:29:19,474][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:29:19,982][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:29:20,480][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:29:20,982][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:29:21,492][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:29:21,990][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:29:22,489][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:29:22,991][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:29:23,493][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:29:23,997][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:29:24,500][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:29:25,003][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:29:25,503][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:29:26,005][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:29:26,507][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:29:27,007][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:29:27,506][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:29:28,006][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:29:28,504][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:29:29,004][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:29:29,502][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:29:30,001][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:29:30,503][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:29:31,001][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:29:31,500][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:29:32,002][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:29:32,501][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:29:32,999][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:29:33,497][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:29:34,000][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:29:34,503][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:29:35,004][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:29:35,507][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:29:36,007][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:29:36,512][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:29:37,018][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:29:37,520][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:29:38,021][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:29:38,524][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:29:39,026][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:29:39,529][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:29:40,032][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:29:40,532][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:29:41,033][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:29:41,539][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:29:42,039][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:29:42,542][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10414 tokens.
[2025-11-13 02:29:43,283][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 02:29:44,065][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:29:44,067][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:29:44,068][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:29:45,106][__main__][INFO] - Iteration 292 took 54s (32.96% Gen, 65.11% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 41m 35s. Estimated total time: 45h 1m 38s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 3s, 500 more iterations: 7h 30m 16s.
[2025-11-13 02:29:45,108][__main__][INFO] - Starting iteration 292.
[2025-11-13 02:29:45,594][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:29:45,594][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:30:03,261][__main__][INFO] - Number of regex retries in iteration 292: 0
[2025-11-13 02:30:03,261][__main__][INFO] - agents played in iteration 292 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:30:04,041][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:30:04,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:30:04,091][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:30:04,113][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:30:04,114][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:30:04,115][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:30:04,817][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:30:05,270][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:30:05,773][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:30:06,271][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:30:06,773][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:30:07,278][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:30:07,777][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:30:08,285][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:30:08,793][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:30:09,293][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:30:09,797][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:30:10,292][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:30:10,791][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:30:11,287][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:30:11,787][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:30:12,291][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:30:12,789][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:30:13,285][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:30:13,796][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:30:14,294][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:30:14,796][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:30:15,300][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:30:15,802][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:30:16,304][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:30:16,804][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:30:17,328][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:30:17,827][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:30:18,329][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:30:18,834][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:30:19,335][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:30:19,837][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:30:20,342][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:30:20,847][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:30:21,347][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:30:21,846][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:30:22,347][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:30:22,851][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:30:23,348][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:30:23,846][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:30:24,343][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:30:24,841][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:30:25,340][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:30:25,836][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:30:26,331][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:30:26,829][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:30:27,327][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:30:27,827][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:30:28,326][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:30:28,827][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:30:29,335][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:30:29,838][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:30:30,340][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:30:30,848][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:30:31,349][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:30:31,853][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:30:32,357][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:30:32,861][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:30:33,379][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:30:33,880][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:30:34,377][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:30:34,879][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:30:35,379][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:30:35,899][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:30:36,396][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:30:36,898][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10361 tokens.
[2025-11-13 02:30:37,631][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.46%, ΔTime: 00:00:32
[2025-11-13 02:30:38,415][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:30:38,417][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:30:38,418][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:30:39,400][__main__][INFO] - Iteration 293 took 53s (32.83% Gen, 65.34% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 29m 22s. Estimated total time: 44h 50m 20s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 40s, 500 more iterations: 7h 28m 23s.
[2025-11-13 02:30:39,402][__main__][INFO] - Starting iteration 293.
[2025-11-13 02:30:39,903][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:30:39,904][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:30:59,265][__main__][INFO] - Number of regex retries in iteration 293: 0
[2025-11-13 02:30:59,266][__main__][INFO] - agents played in iteration 293 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:31:00,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:31:00,128][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:31:00,154][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:31:00,178][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:31:00,179][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:31:00,179][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:31:00,855][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:31:01,307][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:31:01,820][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:31:02,320][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:31:02,819][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:31:03,314][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:31:03,817][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:31:04,313][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:31:04,815][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:31:05,316][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:31:05,814][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:31:06,311][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:31:06,809][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:31:07,310][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:31:07,808][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:31:08,307][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:31:08,808][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:31:09,307][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:31:09,805][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:31:10,304][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:31:10,803][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:31:11,306][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:31:11,805][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:31:12,306][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:31:12,807][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:31:13,309][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:31:13,811][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:31:14,310][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:31:14,813][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:31:15,315][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:31:15,813][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:31:16,310][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:31:16,806][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:31:17,302][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:31:17,806][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:31:18,306][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:31:18,806][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:31:19,302][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:31:19,800][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:31:20,306][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:31:20,806][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:31:21,308][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:31:21,809][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:31:22,309][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:31:22,811][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:31:23,313][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:31:23,812][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:31:24,313][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:31:24,809][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:31:25,310][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:31:25,809][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:31:26,305][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:31:26,807][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:31:27,307][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:31:27,808][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:31:28,307][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:31:28,806][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:31:29,304][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:31:29,804][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:31:30,303][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:31:30,803][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:31:31,302][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:31:31,800][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:31:32,302][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:31:32,805][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10238 tokens.
[2025-11-13 02:31:33,551][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-13 02:31:34,307][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:31:34,308][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:31:34,310][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:31:35,295][__main__][INFO] - Iteration 294 took 55s (34.95% Gen, 63.26% Train). Generation: 19s, Training: 35s. Estimated remaining time: 41h 47m 44s. Estimated total time: 46h 9m 38s. Time estimates for 10 more iterations: 9m 13s, 100 more iterations: 1h 32m 19s, 500 more iterations: 7h 41m 36s.
[2025-11-13 02:31:35,297][__main__][INFO] - Starting iteration 294.
[2025-11-13 02:31:36,056][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:31:36,057][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:31:53,028][__main__][INFO] - Number of regex retries in iteration 294: 0
[2025-11-13 02:31:53,029][__main__][INFO] - agents played in iteration 294 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:31:53,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:31:53,895][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:31:53,921][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:31:53,944][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:31:53,945][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:31:53,946][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:31:54,640][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:31:55,096][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:31:55,607][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:31:56,105][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:31:56,605][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:31:57,108][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:31:57,605][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:31:58,106][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:31:58,614][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:31:59,114][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:31:59,612][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:32:00,108][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:32:00,604][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:32:01,106][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:32:01,606][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:32:02,108][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:32:02,613][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:32:03,112][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:32:03,611][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:32:04,110][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:32:04,610][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:32:05,117][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:32:05,616][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:32:06,116][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:32:06,618][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:32:07,119][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:32:07,625][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:32:08,127][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:32:08,627][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:32:09,131][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:32:09,631][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:32:10,131][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:32:10,634][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:32:11,135][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:32:11,632][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:32:12,132][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:32:12,634][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:32:13,137][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:32:13,636][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:32:14,139][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:32:14,639][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:32:15,138][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:32:15,640][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:32:16,136][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:32:16,636][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:32:17,136][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:32:17,632][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:32:18,132][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:32:18,633][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:32:19,135][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:32:19,638][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:32:20,139][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:32:20,640][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:32:21,143][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:32:21,646][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:32:22,151][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:32:22,653][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:32:23,154][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:32:23,655][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:32:24,156][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:32:24,658][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:32:25,159][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:32:25,658][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:32:26,160][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:32:26,663][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10412 tokens.
[2025-11-13 02:32:27,410][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 02:32:28,158][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:32:28,160][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:32:28,161][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:32:29,304][__main__][INFO] - Iteration 295 took 53s (31.87% Gen, 65.98% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 59m 39s. Estimated total time: 44h 22m 27s. Time estimates for 10 more iterations: 8m 52s, 100 more iterations: 1h 28m 44s, 500 more iterations: 7h 23m 44s.
[2025-11-13 02:32:29,307][__main__][INFO] - Starting iteration 295.
[2025-11-13 02:32:29,806][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:32:29,806][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:32:34,526][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:32:47,498][__main__][INFO] - Number of regex retries in iteration 295: 1
[2025-11-13 02:32:47,498][__main__][INFO] - agents played in iteration 295 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:32:48,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:32:48,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:32:48,353][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:32:48,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:32:48,376][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:32:48,376][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:32:49,066][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:32:49,524][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:32:50,031][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:32:50,529][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:32:51,027][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:32:51,532][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:32:52,030][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:32:52,533][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:32:53,030][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:32:53,528][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:32:54,029][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:32:54,534][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:32:55,039][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:32:55,540][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:32:56,041][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:32:56,541][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:32:57,044][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:32:57,547][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:32:58,048][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:32:58,549][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:32:59,048][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:32:59,546][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:33:00,046][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:33:00,546][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:33:01,049][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:33:01,547][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:33:02,048][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:33:02,552][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:33:03,051][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:33:03,550][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:33:04,048][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:33:04,548][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:33:05,047][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:33:05,550][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:33:06,051][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:33:06,550][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:33:07,046][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:33:07,543][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:33:08,043][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:33:08,539][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:33:09,035][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:33:09,532][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:33:10,031][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:33:10,528][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:33:11,026][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:33:11,527][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:33:12,024][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:33:12,527][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:33:13,026][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:33:13,525][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:33:14,029][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:33:14,531][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:33:15,034][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:33:15,535][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:33:16,039][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:33:16,541][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:33:17,042][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:33:17,541][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:33:18,045][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:33:18,547][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:33:19,049][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:33:19,549][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:33:20,046][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:33:20,542][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:33:21,040][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10337 tokens.
[2025-11-13 02:33:21,753][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.13%, ΔTime: 00:00:32
[2025-11-13 02:33:22,514][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:33:22,515][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:33:22,517][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:33:23,543][__main__][INFO] - Iteration 296 took 53s (32.92% Gen, 65.17% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 23m 10s. Estimated total time: 44h 46m 51s. Time estimates for 10 more iterations: 8m 57s, 100 more iterations: 1h 29m 33s, 500 more iterations: 7h 27m 48s.
[2025-11-13 02:33:23,545][__main__][INFO] - Starting iteration 296.
[2025-11-13 02:33:24,039][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:33:24,039][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:33:42,060][__main__][INFO] - Number of regex retries in iteration 296: 0
[2025-11-13 02:33:42,061][__main__][INFO] - agents played in iteration 296 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:33:42,841][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:33:42,866][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:33:42,890][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:33:42,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:33:42,913][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:33:42,914][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:33:43,612][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:33:44,072][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:33:44,572][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:33:45,071][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:33:45,579][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:33:46,082][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:33:46,584][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:33:47,085][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:33:47,589][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:33:48,086][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:33:48,585][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:33:49,095][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:33:49,592][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:33:50,096][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:33:50,595][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:33:51,096][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:33:51,600][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:33:52,109][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:33:52,624][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:33:53,128][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:33:53,635][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:33:54,140][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:33:54,642][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:33:55,145][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:33:55,649][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:33:56,149][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:33:56,657][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:33:57,156][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:33:57,657][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:33:58,159][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:33:58,659][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:33:59,157][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:33:59,658][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:34:00,155][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:34:00,656][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:34:01,154][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:34:01,652][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:34:02,152][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:34:02,649][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:34:03,146][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:34:03,644][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:34:04,142][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:34:04,643][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:34:05,143][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:34:05,642][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:34:06,145][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:34:06,644][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:34:07,146][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:34:07,648][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:34:08,148][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:34:08,649][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:34:09,147][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:34:09,647][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:34:10,148][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:34:10,648][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:34:11,146][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:34:11,643][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:34:12,144][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:34:12,647][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:34:13,149][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:34:13,653][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:34:14,151][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:34:14,649][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:34:15,151][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:34:15,652][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10312 tokens.
[2025-11-13 02:34:16,359][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.36%, ΔTime: 00:00:32
[2025-11-13 02:34:17,114][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:34:17,116][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:34:17,118][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:34:18,092][__main__][INFO] - Iteration 297 took 54s (33.34% Gen, 64.86% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 38m 3s. Estimated total time: 45h 2m 39s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 5s, 500 more iterations: 7h 30m 26s.
[2025-11-13 02:34:18,094][__main__][INFO] - Starting iteration 297.
[2025-11-13 02:34:18,599][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:34:18,600][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:34:24,321][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:34:34,963][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:34:37,821][__main__][INFO] - Number of regex retries in iteration 297: 2
[2025-11-13 02:34:37,822][__main__][INFO] - agents played in iteration 297 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:34:38,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:34:38,657][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:34:38,683][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:34:38,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:34:38,707][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:34:38,707][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:34:39,393][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:34:39,850][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:34:40,358][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:34:40,856][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:34:41,354][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:34:41,857][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:34:42,359][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:34:42,858][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:34:43,362][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:34:43,864][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:34:44,367][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:34:44,869][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:34:45,377][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:34:45,879][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:34:46,383][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:34:46,886][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:34:47,390][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:34:47,897][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:34:48,397][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:34:48,897][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:34:49,396][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:34:49,898][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:34:50,396][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:34:50,895][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:34:51,396][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:34:51,905][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:34:52,405][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:34:52,904][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:34:53,407][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:34:53,905][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:34:54,406][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:34:54,907][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:34:55,407][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:34:55,925][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:34:56,424][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:34:56,923][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:34:57,424][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:34:57,924][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:34:58,430][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:34:58,930][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:34:59,433][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:34:59,933][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:35:00,430][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:35:00,933][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:35:01,431][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:35:01,930][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:35:02,440][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:35:02,935][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:35:03,434][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:35:03,933][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:35:04,431][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:35:04,946][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:35:05,441][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:35:05,940][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:35:06,441][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:35:06,941][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:35:07,445][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:35:07,941][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:35:08,435][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:35:08,932][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:35:09,427][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:35:09,930][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:35:10,425][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:35:10,919][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:35:11,416][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10279 tokens.
[2025-11-13 02:35:12,108][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 02:35:12,880][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:35:12,881][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:35:12,883][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:35:13,900][__main__][INFO] - Iteration 298 took 55s (34.76% Gen, 63.40% Train). Generation: 19s, Training: 35s. Estimated remaining time: 41h 39m 30s. Estimated total time: 46h 5m 2s. Time estimates for 10 more iterations: 9m 13s, 100 more iterations: 1h 32m 10s, 500 more iterations: 7h 40m 50s.
[2025-11-13 02:35:13,902][__main__][INFO] - Starting iteration 298.
[2025-11-13 02:35:14,381][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:35:14,382][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:35:25,804][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:35:32,106][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:35:33,058][__main__][INFO] - Number of regex retries in iteration 298: 2
[2025-11-13 02:35:33,058][__main__][INFO] - agents played in iteration 298 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:35:33,848][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:35:33,875][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:35:33,902][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:35:33,924][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:35:33,925][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:35:33,926][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:35:34,609][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:35:35,074][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:35:35,584][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:35:36,091][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:35:36,598][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:35:37,101][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:35:37,604][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:35:38,108][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:35:38,611][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:35:39,125][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:35:39,626][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:35:40,142][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:35:40,641][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:35:41,142][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:35:41,644][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:35:42,144][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:35:42,648][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:35:43,149][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:35:43,648][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:35:44,150][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:35:44,653][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:35:45,161][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:35:45,660][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:35:46,160][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:35:46,661][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:35:47,162][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:35:47,661][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:35:48,162][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:35:48,664][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:35:49,163][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:35:49,666][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:35:50,167][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:35:50,672][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:35:51,175][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:35:51,682][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:35:52,184][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:35:52,691][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:35:53,198][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:35:53,699][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:35:54,203][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:35:54,706][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:35:55,206][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:35:55,704][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:35:56,203][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:35:56,700][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:35:57,201][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:35:57,697][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:35:58,199][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:35:58,703][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:35:59,200][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:35:59,698][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:36:00,200][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:36:00,701][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:36:01,204][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:36:01,703][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:36:02,198][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:36:02,695][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:36:03,191][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:36:03,691][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:36:04,188][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:36:04,688][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:36:05,186][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:36:05,687][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:36:06,184][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:36:06,683][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10320 tokens.
[2025-11-13 02:36:07,376][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 02:36:08,150][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:36:08,151][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:36:08,153][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:36:09,106][__main__][INFO] - Iteration 299 took 54s (34.13% Gen, 64.13% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 9m 49s. Estimated total time: 45h 36m 16s. Time estimates for 10 more iterations: 9m 7s, 100 more iterations: 1h 31m 12s, 500 more iterations: 7h 36m 2s.
[2025-11-13 02:36:09,108][__main__][INFO] - Starting iteration 299.
[2025-11-13 02:36:09,619][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:36:09,619][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:36:14,998][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:36:17,438][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:36:27,678][__main__][INFO] - Number of regex retries in iteration 299: 2
[2025-11-13 02:36:27,679][__main__][INFO] - agents played in iteration 299 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:36:28,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:36:28,556][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:36:28,578][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:36:28,601][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:36:28,601][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:36:28,602][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:36:29,355][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:36:29,816][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:36:30,321][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:36:30,828][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:36:31,330][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:36:31,831][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:36:32,334][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:36:32,838][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:36:33,338][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:36:33,849][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:36:34,352][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:36:34,854][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:36:35,360][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:36:35,863][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:36:36,372][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:36:36,876][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:36:37,381][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:36:37,882][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:36:38,387][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:36:38,890][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:36:39,393][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:36:39,896][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:36:40,397][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:36:40,897][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:36:41,394][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:36:41,893][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:36:42,394][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:36:42,897][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:36:43,397][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:36:43,894][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:36:44,396][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:36:44,896][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:36:45,397][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:36:45,899][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:36:46,400][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:36:46,901][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:36:47,398][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:36:47,895][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:36:48,394][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:36:48,894][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:36:49,393][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:36:49,894][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:36:50,392][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:36:50,890][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:36:51,391][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:36:51,887][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:36:52,390][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:36:52,892][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:36:53,390][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:36:53,891][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:36:54,390][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:36:54,891][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:36:55,392][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:36:55,894][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:36:56,394][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:36:56,892][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:36:57,390][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:36:57,888][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:36:58,386][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:36:58,883][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:36:59,379][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:36:59,875][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:37:00,374][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:37:00,873][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:37:01,369][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10401 tokens.
[2025-11-13 02:37:02,055][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 02:37:02,839][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:37:02,841][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:37:02,843][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:37:03,771][__main__][INFO] - Iteration 300 took 54s (33.35% Gen, 64.93% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 40m 16s. Estimated total time: 45h 7m 38s. Time estimates for 10 more iterations: 9m 1s, 100 more iterations: 1h 30m 15s, 500 more iterations: 7h 31m 16s.
[2025-11-13 02:37:03,773][__main__][INFO] - Starting iteration 300.
[2025-11-13 02:37:04,264][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 02:37:04,264][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:37:23,709][__main__][INFO] - Number of regex retries in iteration 300: 0
[2025-11-13 02:37:23,710][__main__][INFO] - agents played in iteration 300 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:37:24,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:37:24,527][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:37:24,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:37:24,577][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:37:24,578][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:37:24,579][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:37:25,274][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:37:25,727][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:37:26,228][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:37:26,737][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:37:27,238][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:37:27,754][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:37:28,251][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:37:28,753][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:37:29,267][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:37:29,766][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:37:30,274][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:37:30,801][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:37:31,306][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:37:31,811][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:37:32,313][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:37:32,816][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:37:33,329][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:37:33,830][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:37:34,332][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:37:34,833][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:37:35,332][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:37:35,859][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:37:36,356][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:37:36,852][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:37:37,353][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:37:37,850][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:37:38,358][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:37:38,855][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:37:39,350][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:37:39,866][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:37:40,365][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:37:40,862][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:37:41,359][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:37:41,860][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:37:42,377][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:37:42,884][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:37:43,388][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:37:43,888][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:37:44,386][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:37:44,888][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:37:45,386][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:37:45,886][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:37:46,389][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:37:46,890][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:37:47,390][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:37:47,885][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:37:48,386][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:37:48,886][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:37:49,388][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:37:49,889][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:37:50,386][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:37:50,884][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:37:51,382][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:37:51,882][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:37:52,380][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:37:52,886][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:37:53,385][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:37:53,881][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:37:54,374][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:37:54,872][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:37:55,372][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:37:55,871][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:37:56,366][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:37:56,868][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:37:57,365][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10392 tokens.
[2025-11-13 02:37:58,063][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 02:37:58,815][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:37:58,817][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:37:58,819][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:38:00,832][__main__][INFO] - Iteration 301 took 56s (34.38% Gen, 62.06% Train). Generation: 19s, Training: 35s. Estimated remaining time: 42h 40m 7s. Estimated total time: 47h 8m 26s. Time estimates for 10 more iterations: 9m 25s, 100 more iterations: 1h 34m 16s, 500 more iterations: 7h 51m 24s.
[2025-11-13 02:38:00,834][__main__][INFO] - Starting iteration 301.
[2025-11-13 02:38:01,318][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:38:01,318][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:38:12,661][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:38:20,824][__main__][INFO] - Number of regex retries in iteration 301: 1
[2025-11-13 02:38:20,824][__main__][INFO] - agents played in iteration 301 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:38:21,632][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:38:21,660][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:38:21,686][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:38:21,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:38:21,711][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:38:21,712][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:38:22,441][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:38:22,906][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:38:23,412][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:38:23,914][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:38:24,423][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:38:24,926][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:38:25,430][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:38:25,933][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:38:26,439][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:38:26,947][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:38:27,456][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:38:27,961][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:38:28,471][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:38:28,969][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:38:29,480][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:38:29,979][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:38:30,477][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:38:30,974][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:38:31,473][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:38:31,968][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:38:32,472][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:38:32,970][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:38:33,490][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:38:33,987][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:38:34,484][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:38:34,987][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:38:35,486][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:38:35,987][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:38:36,486][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:38:36,987][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:38:37,502][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:38:38,002][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:38:38,504][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:38:38,999][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:38:39,497][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:38:40,003][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:38:40,504][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:38:41,002][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:38:41,507][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:38:42,007][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:38:42,510][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:38:43,013][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:38:43,509][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:38:44,008][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:38:44,502][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:38:44,998][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:38:45,505][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:38:46,002][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:38:46,501][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:38:46,999][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:38:47,494][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:38:47,998][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:38:48,492][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:38:48,988][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:38:49,487][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:38:49,983][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:38:50,490][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:38:50,987][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:38:51,482][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:38:51,978][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:38:52,477][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:38:52,971][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:38:53,466][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:38:53,960][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:38:54,455][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10339 tokens.
[2025-11-13 02:38:55,156][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 02:38:55,922][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:38:55,924][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:38:55,926][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:38:56,831][__main__][INFO] - Iteration 302 took 55s (35.14% Gen, 63.23% Train). Generation: 19s, Training: 35s. Estimated remaining time: 41h 46m 25s. Estimated total time: 46h 15m 41s. Time estimates for 10 more iterations: 9m 15s, 100 more iterations: 1h 32m 31s, 500 more iterations: 7h 42m 36s.
[2025-11-13 02:38:56,833][__main__][INFO] - Starting iteration 302.
[2025-11-13 02:38:57,304][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:38:57,305][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:39:05,458][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:39:12,752][__main__][INFO] - Number of regex retries in iteration 302: 1
[2025-11-13 02:39:12,752][__main__][INFO] - agents played in iteration 302 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:39:13,551][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:39:13,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:39:13,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:39:13,618][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:39:13,618][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:39:13,619][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:39:14,327][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:39:14,784][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:39:15,291][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:39:15,793][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:39:16,296][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:39:16,802][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:39:17,305][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:39:17,811][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:39:18,314][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:39:18,819][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:39:19,321][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:39:19,825][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:39:20,325][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:39:20,827][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:39:21,337][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:39:21,836][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:39:22,337][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:39:22,836][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:39:23,330][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:39:23,828][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:39:24,328][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:39:24,825][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:39:25,321][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:39:25,821][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:39:26,319][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:39:26,818][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:39:27,318][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:39:27,816][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:39:28,315][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:39:28,815][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:39:29,313][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:39:29,811][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:39:30,313][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:39:30,811][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:39:31,311][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:39:31,806][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:39:32,305][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:39:32,807][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:39:33,332][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:39:33,836][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:39:34,337][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:39:34,836][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:39:35,342][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:39:35,840][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:39:36,344][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:39:36,851][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:39:37,353][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:39:37,854][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:39:38,352][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:39:38,847][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:39:39,348][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:39:39,843][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:39:40,339][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:39:40,835][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:39:41,331][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:39:41,829][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:39:42,326][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:39:42,822][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:39:43,321][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:39:43,816][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:39:44,317][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:39:44,813][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:39:45,310][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:39:45,809][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:39:46,306][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10245 tokens.
[2025-11-13 02:39:46,999][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 02:39:47,755][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:39:47,757][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:39:47,758][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:39:48,673][__main__][INFO] - Iteration 303 took 51s (30.07% Gen, 68.15% Train). Generation: 15s, Training: 35s. Estimated remaining time: 38h 18m 20s. Estimated total time: 42h 48m 27s. Time estimates for 10 more iterations: 8m 33s, 100 more iterations: 1h 25m 36s, 500 more iterations: 7h 8m 4s.
[2025-11-13 02:39:48,675][__main__][INFO] - Starting iteration 303.
[2025-11-13 02:39:49,148][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:39:49,148][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:40:04,212][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:40:05,515][__main__][INFO] - Number of regex retries in iteration 303: 1
[2025-11-13 02:40:05,516][__main__][INFO] - agents played in iteration 303 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:40:06,301][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:40:06,325][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:40:06,350][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:40:06,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:40:06,373][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:40:06,375][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:40:07,090][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:40:07,541][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:40:08,047][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:40:08,553][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:40:09,054][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:40:09,561][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:40:10,061][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:40:10,560][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:40:11,061][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:40:11,562][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:40:12,078][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:40:12,577][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:40:13,076][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:40:13,577][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:40:14,077][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:40:14,598][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:40:15,098][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:40:15,602][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:40:16,106][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:40:16,609][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:40:17,108][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:40:17,610][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:40:18,110][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:40:18,609][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:40:19,110][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:40:19,609][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:40:20,114][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:40:20,614][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:40:21,115][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:40:21,613][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:40:22,112][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:40:22,619][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:40:23,119][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:40:23,617][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:40:24,120][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:40:24,617][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:40:25,113][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:40:25,610][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:40:26,108][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:40:26,612][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:40:27,110][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:40:27,608][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:40:28,104][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:40:28,600][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:40:29,106][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:40:29,610][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:40:30,107][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:40:30,615][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:40:31,111][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:40:31,613][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:40:32,119][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:40:32,623][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:40:33,127][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:40:33,625][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:40:34,124][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:40:34,620][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:40:35,118][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:40:35,619][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:40:36,119][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:40:36,618][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:40:37,118][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:40:37,615][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:40:38,109][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:40:38,616][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:40:39,112][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10191 tokens.
[2025-11-13 02:40:39,828][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 02:40:40,599][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:40:40,601][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:40:40,603][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:40:41,641][__main__][INFO] - Iteration 304 took 52s (31.18% Gen, 66.84% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 13m 42s. Estimated total time: 43h 44m 42s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 29s, 500 more iterations: 7h 17m 27s.
[2025-11-13 02:40:41,643][__main__][INFO] - Starting iteration 304.
[2025-11-13 02:40:42,145][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:40:42,146][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:40:54,838][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:41:00,419][__main__][INFO] - Number of regex retries in iteration 304: 1
[2025-11-13 02:41:00,420][__main__][INFO] - agents played in iteration 304 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:41:01,250][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:41:01,277][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:41:01,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:41:01,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:41:01,327][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:41:01,327][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:41:02,064][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:41:02,523][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:41:03,024][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:41:03,526][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:41:04,022][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:41:04,518][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:41:05,019][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:41:05,519][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:41:06,020][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:41:06,525][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:41:07,024][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:41:07,525][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:41:08,026][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:41:08,530][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:41:09,031][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:41:09,530][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:41:10,033][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:41:10,531][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:41:11,029][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:41:11,533][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:41:12,032][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:41:12,530][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:41:13,031][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:41:13,529][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:41:14,029][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:41:14,529][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:41:15,029][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:41:15,529][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:41:16,027][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:41:16,526][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:41:17,023][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:41:17,521][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:41:18,022][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:41:18,519][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:41:19,015][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:41:19,516][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:41:20,011][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:41:20,509][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:41:21,009][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:41:21,506][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:41:22,017][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:41:22,513][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:41:23,011][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:41:23,510][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:41:24,009][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:41:24,513][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:41:25,012][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:41:25,511][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:41:26,023][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:41:26,521][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:41:27,019][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:41:27,518][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:41:28,017][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:41:28,524][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:41:29,027][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:41:29,526][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:41:30,031][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:41:30,528][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:41:31,029][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:41:31,530][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:41:32,026][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:41:32,529][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:41:33,026][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:41:33,526][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:41:34,029][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10263 tokens.
[2025-11-13 02:41:34,766][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 02:41:35,564][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:41:35,566][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:41:35,568][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:41:36,514][__main__][INFO] - Iteration 305 took 54s (33.61% Gen, 64.65% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 46m 34s. Estimated total time: 45h 18m 29s. Time estimates for 10 more iterations: 9m 3s, 100 more iterations: 1h 30m 36s, 500 more iterations: 7h 33m 4s.
[2025-11-13 02:41:36,516][__main__][INFO] - Starting iteration 305.
[2025-11-13 02:41:37,013][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:41:37,015][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:41:48,479][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 30 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:41:55,310][__main__][INFO] - Number of regex retries in iteration 305: 1
[2025-11-13 02:41:55,311][__main__][INFO] - agents played in iteration 305 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:41:56,301][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:41:56,324][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:41:56,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:41:56,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:41:56,369][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:41:56,369][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:41:57,059][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:41:57,517][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:41:58,020][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:41:58,519][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:41:59,025][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:41:59,522][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:42:00,021][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:42:00,523][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:42:01,022][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:42:01,536][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:42:02,037][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:42:02,541][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:42:03,046][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:42:03,546][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:42:04,047][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:42:04,544][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:42:05,044][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:42:05,543][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:42:06,046][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:42:06,549][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:42:07,050][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:42:07,553][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:42:08,054][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:42:08,557][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:42:09,053][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:42:09,553][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:42:10,050][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:42:10,546][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:42:11,046][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:42:11,543][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:42:12,052][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:42:12,547][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:42:13,046][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:42:13,545][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:42:14,045][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:42:14,547][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:42:15,048][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:42:15,547][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:42:16,048][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:42:16,565][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:42:17,067][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:42:17,564][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:42:18,064][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:42:18,564][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:42:19,061][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:42:19,558][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:42:20,055][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:42:20,553][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:42:21,055][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:42:21,551][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:42:22,045][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:42:22,546][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:42:23,046][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:42:23,547][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:42:24,046][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:42:24,543][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:42:25,048][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:42:25,546][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:42:26,044][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:42:26,541][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:42:27,038][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:42:27,542][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:42:28,038][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:42:28,538][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:42:29,043][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10224 tokens.
[2025-11-13 02:42:29,822][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:32
[2025-11-13 02:42:30,610][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:42:30,611][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:42:30,613][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:42:31,600][__main__][INFO] - Iteration 306 took 54s (33.52% Gen, 64.67% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 56m 31s. Estimated total time: 45h 29m 21s. Time estimates for 10 more iterations: 9m 5s, 100 more iterations: 1h 30m 58s, 500 more iterations: 7h 34m 53s.
[2025-11-13 02:42:31,602][__main__][INFO] - Starting iteration 306.
[2025-11-13 02:42:32,080][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:42:32,081][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:42:41,436][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 20 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:42:47,614][__main__][INFO] - Number of regex retries in iteration 306: 1
[2025-11-13 02:42:47,615][__main__][INFO] - agents played in iteration 306 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:42:48,448][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:42:48,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:42:48,514][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:42:48,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:42:48,539][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:42:48,540][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:42:49,226][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:42:49,679][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:42:50,181][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:42:50,687][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:42:51,187][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:42:51,689][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:42:52,187][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:42:52,691][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:42:53,189][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:42:53,687][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:42:54,185][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:42:54,684][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:42:55,186][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:42:55,689][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:42:56,189][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:42:56,688][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:42:57,188][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:42:57,691][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:42:58,189][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:42:58,689][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:42:59,193][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:42:59,695][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:43:00,196][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:43:00,698][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:43:01,197][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:43:01,704][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:43:02,206][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:43:02,711][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:43:03,225][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:43:03,726][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:43:04,250][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:43:04,753][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:43:05,253][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:43:05,753][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:43:06,252][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:43:06,751][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:43:07,249][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:43:07,748][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:43:08,246][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:43:08,745][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:43:09,244][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:43:09,756][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:43:10,252][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:43:10,755][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:43:11,251][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:43:11,746][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:43:12,259][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:43:12,760][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:43:13,256][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:43:13,764][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:43:14,263][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:43:14,761][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:43:15,256][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:43:15,752][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:43:16,255][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:43:16,752][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:43:17,249][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:43:17,746][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:43:18,245][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:43:18,747][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:43:19,246][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:43:19,749][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:43:20,261][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:43:20,761][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:43:21,262][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10296 tokens.
[2025-11-13 02:43:22,027][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 02:43:22,798][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:43:22,799][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:43:22,801][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:43:23,652][__main__][INFO] - Iteration 307 took 51s (30.12% Gen, 68.23% Train). Generation: 15s, Training: 35s. Estimated remaining time: 38h 24m 53s. Estimated total time: 42h 58m 35s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 57s, 500 more iterations: 7h 9m 45s.
[2025-11-13 02:43:23,654][__main__][INFO] - Starting iteration 307.
[2025-11-13 02:43:24,135][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:43:24,136][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:43:28,889][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:43:33,253][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:43:42,049][__main__][INFO] - Number of regex retries in iteration 307: 2
[2025-11-13 02:43:42,050][__main__][INFO] - agents played in iteration 307 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:43:42,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:43:42,938][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:43:42,965][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:43:42,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:43:42,988][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:43:42,989][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:43:43,679][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:43:44,151][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:43:44,655][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:43:45,156][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:43:45,655][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:43:46,155][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:43:46,665][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:43:47,165][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:43:47,664][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:43:48,160][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:43:48,656][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:43:49,156][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:43:49,653][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:43:50,153][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:43:50,661][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:43:51,162][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:43:51,663][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:43:52,159][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:43:52,661][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:43:53,167][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:43:53,665][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:43:54,167][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:43:54,667][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:43:55,167][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:43:55,669][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:43:56,172][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:43:56,676][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:43:57,179][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:43:57,683][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:43:58,188][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:43:58,688][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:43:59,190][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:43:59,690][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:44:00,192][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:44:00,694][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:44:01,194][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:44:01,693][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:44:02,193][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:44:02,694][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:44:03,190][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:44:03,694][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:44:04,194][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:44:04,692][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:44:05,190][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:44:05,691][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:44:06,192][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:44:06,690][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:44:07,190][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:44:07,688][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:44:08,185][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:44:08,687][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:44:09,185][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:44:09,684][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:44:10,184][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:44:10,685][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:44:11,186][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:44:11,687][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:44:12,190][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:44:12,696][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:44:13,201][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:44:13,709][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:44:14,224][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:44:14,725][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:44:15,228][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:44:15,729][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10368 tokens.
[2025-11-13 02:44:16,483][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 02:44:17,216][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:44:17,218][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:44:17,220][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:44:18,178][__main__][INFO] - Iteration 308 took 54s (33.15% Gen, 65.08% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 27m 33s. Estimated total time: 45h 2m 10s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 4s, 500 more iterations: 7h 30m 21s.
[2025-11-13 02:44:18,180][__main__][INFO] - Starting iteration 308.
[2025-11-13 02:44:18,676][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:44:18,676][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:44:34,950][__main__][INFO] - Number of regex retries in iteration 308: 0
[2025-11-13 02:44:34,951][__main__][INFO] - agents played in iteration 308 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:44:35,777][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:44:35,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:44:35,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:44:35,855][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:44:35,856][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:44:35,856][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:44:36,575][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:44:37,029][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:44:37,537][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:44:38,038][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:44:38,535][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:44:39,046][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:44:39,545][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:44:40,042][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:44:40,539][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:44:41,042][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:44:41,551][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:44:42,049][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:44:42,550][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:44:43,055][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:44:43,554][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:44:44,068][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:44:44,569][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:44:45,069][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:44:45,578][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:44:46,081][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:44:46,614][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:44:47,117][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:44:47,620][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:44:48,125][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:44:48,632][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:44:49,132][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:44:49,635][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:44:50,136][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:44:50,640][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:44:51,142][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:44:51,643][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:44:52,160][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:44:52,660][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:44:53,163][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:44:53,663][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:44:54,161][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:44:54,669][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:44:55,167][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:44:55,669][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:44:56,166][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:44:56,665][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:44:57,166][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:44:57,666][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:44:58,164][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:44:58,672][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:44:59,168][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:44:59,665][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:45:00,167][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:45:00,668][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:45:01,169][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:45:01,667][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:45:02,166][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:45:02,682][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:45:03,180][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:45:03,678][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:45:04,181][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:45:04,678][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:45:05,178][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:45:05,681][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:45:06,180][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:45:06,691][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:45:07,194][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:45:07,693][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:45:08,199][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:45:08,700][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10256 tokens.
[2025-11-13 02:45:09,455][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 02:45:10,188][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:45:10,190][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:45:10,191][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:45:11,123][__main__][INFO] - Iteration 309 took 52s (31.03% Gen, 67.19% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 6m 54s. Estimated total time: 43h 42m 23s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 24s, 500 more iterations: 7h 17m 3s.
[2025-11-13 02:45:11,126][__main__][INFO] - Starting iteration 309.
[2025-11-13 02:45:11,616][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:45:11,617][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:45:30,192][__main__][INFO] - Number of regex retries in iteration 309: 0
[2025-11-13 02:45:30,192][__main__][INFO] - agents played in iteration 309 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:45:31,008][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:45:31,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:45:31,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:45:31,077][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:45:31,077][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:45:31,078][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:45:31,799][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:45:32,253][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:45:32,754][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:45:33,255][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:45:33,754][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:45:34,254][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:45:34,752][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:45:35,249][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:45:35,747][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:45:36,245][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:45:36,742][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:45:37,249][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:45:37,748][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:45:38,272][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:45:38,770][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:45:39,265][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:45:39,765][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:45:40,261][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:45:40,775][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:45:41,273][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:45:41,771][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:45:42,285][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:45:42,783][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:45:43,287][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:45:43,789][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:45:44,293][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:45:44,796][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:45:45,296][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:45:45,805][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:45:46,309][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:45:46,810][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:45:47,307][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:45:47,805][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:45:48,302][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:45:48,804][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:45:49,300][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:45:49,796][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:45:50,294][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:45:50,794][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:45:51,293][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:45:51,788][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:45:52,285][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:45:52,781][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:45:53,275][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:45:53,784][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:45:54,279][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:45:54,777][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:45:55,274][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:45:55,771][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:45:56,270][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:45:56,771][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:45:57,271][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:45:57,785][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:45:58,282][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:45:58,783][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:45:59,287][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:45:59,785][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:46:00,290][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:46:00,793][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:46:01,292][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:46:01,793][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:46:02,292][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:46:02,789][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:46:03,289][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:46:03,789][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10264 tokens.
[2025-11-13 02:46:04,544][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 02:46:05,286][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:46:05,287][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:46:05,290][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:46:06,263][__main__][INFO] - Iteration 310 took 54s (33.99% Gen, 64.22% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 55m 57s. Estimated total time: 45h 32m 22s. Time estimates for 10 more iterations: 9m 6s, 100 more iterations: 1h 31m 4s, 500 more iterations: 7h 35m 23s.
[2025-11-13 02:46:06,265][__main__][INFO] - Starting iteration 310.
[2025-11-13 02:46:06,757][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 02:46:06,758][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:46:25,287][__main__][INFO] - Number of regex retries in iteration 310: 0
[2025-11-13 02:46:25,288][__main__][INFO] - agents played in iteration 310 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:46:26,161][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:46:26,184][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:46:26,206][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:46:26,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:46:26,228][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:46:26,229][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:46:26,930][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:46:27,385][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:46:27,888][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:46:28,388][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:46:28,889][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:46:29,390][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:46:29,889][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:46:30,388][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:46:30,906][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:46:31,404][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:46:31,900][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:46:32,396][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:46:32,894][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:46:33,399][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:46:33,901][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:46:34,400][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:46:34,905][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:46:35,409][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:46:35,911][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:46:36,417][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:46:36,919][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:46:37,421][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:46:37,924][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:46:38,430][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:46:38,935][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:46:39,433][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:46:39,936][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:46:40,438][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:46:40,942][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:46:41,442][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:46:41,944][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:46:42,442][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:46:42,943][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:46:43,440][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:46:43,937][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:46:44,434][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:46:44,931][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:46:45,431][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:46:45,927][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:46:46,425][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:46:46,924][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:46:47,420][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:46:47,917][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:46:48,412][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:46:48,909][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:46:49,413][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:46:49,909][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:46:50,407][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:46:50,907][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:46:51,408][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:46:51,914][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:46:52,428][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:46:52,947][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:46:53,452][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:46:53,953][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:46:54,457][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:46:54,960][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:46:55,459][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:46:55,961][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:46:56,460][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:46:56,959][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:46:57,465][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:46:57,966][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:46:58,475][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:46:58,970][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10231 tokens.
[2025-11-13 02:46:59,684][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 02:47:00,446][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:47:00,447][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:47:00,449][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:47:02,267][__main__][INFO] - Iteration 311 took 55s (33.38% Gen, 63.34% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 38m 8s. Estimated total time: 46h 15m 29s. Time estimates for 10 more iterations: 9m 15s, 100 more iterations: 1h 32m 30s, 500 more iterations: 7h 42m 34s.
[2025-11-13 02:47:02,269][__main__][INFO] - Starting iteration 311.
[2025-11-13 02:47:02,734][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:47:02,736][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:47:19,591][__main__][INFO] - Number of regex retries in iteration 311: 0
[2025-11-13 02:47:19,591][__main__][INFO] - agents played in iteration 311 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:47:20,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:47:20,457][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:47:20,480][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:47:20,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:47:20,503][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:47:20,504][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:47:21,242][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:47:21,702][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:47:22,223][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:47:22,723][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:47:23,225][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:47:23,724][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:47:24,223][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:47:24,726][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:47:25,224][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:47:25,722][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:47:26,219][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:47:26,717][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:47:27,215][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:47:27,714][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:47:28,214][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:47:28,725][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:47:29,226][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:47:29,728][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:47:30,226][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:47:30,728][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:47:31,233][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:47:31,734][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:47:32,236][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:47:32,738][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:47:33,249][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:47:33,755][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:47:34,265][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:47:34,763][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:47:35,265][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:47:35,760][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:47:36,262][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:47:36,760][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:47:37,260][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:47:37,758][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:47:38,256][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:47:38,755][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:47:39,261][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:47:39,761][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:47:40,260][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:47:40,761][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:47:41,259][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:47:41,755][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:47:42,251][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:47:42,747][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:47:43,245][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:47:43,743][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:47:44,242][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:47:44,743][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:47:45,247][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:47:45,747][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:47:46,246][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:47:46,746][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:47:47,249][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:47:47,755][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:47:48,257][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:47:48,758][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:47:49,256][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:47:49,759][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:47:50,260][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:47:50,762][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:47:51,265][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:47:51,763][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:47:52,262][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:47:52,765][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:47:53,263][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10184 tokens.
[2025-11-13 02:47:53,992][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 02:47:54,751][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:47:54,753][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:47:54,754][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:47:55,677][__main__][INFO] - Iteration 312 took 52s (31.84% Gen, 66.42% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 28m 54s. Estimated total time: 44h 7m 8s. Time estimates for 10 more iterations: 8m 49s, 100 more iterations: 1h 28m 14s, 500 more iterations: 7h 21m 11s.
[2025-11-13 02:47:55,679][__main__][INFO] - Starting iteration 312.
[2025-11-13 02:47:56,183][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:47:56,184][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:48:12,954][__main__][INFO] - Number of regex retries in iteration 312: 0
[2025-11-13 02:48:12,955][__main__][INFO] - agents played in iteration 312 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:48:13,804][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:48:13,831][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:48:13,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:48:13,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:48:13,879][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:48:13,880][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:48:14,600][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:48:15,053][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:48:15,558][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:48:16,057][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:48:16,553][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:48:17,050][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:48:17,547][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:48:18,044][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:48:18,546][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:48:19,051][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:48:19,548][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:48:20,048][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:48:20,546][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:48:21,047][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:48:21,553][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:48:22,054][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:48:22,557][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:48:23,061][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:48:23,562][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:48:24,066][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:48:24,573][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:48:25,072][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:48:25,571][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:48:26,069][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:48:26,569][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:48:27,068][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:48:27,569][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:48:28,073][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:48:28,574][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:48:29,075][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:48:29,574][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:48:30,074][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:48:30,576][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:48:31,074][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:48:31,573][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:48:32,081][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:48:32,579][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:48:33,076][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:48:33,571][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:48:34,067][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:48:34,566][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:48:35,068][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:48:35,571][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:48:36,073][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:48:36,574][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:48:37,078][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:48:37,578][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:48:38,076][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:48:38,583][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:48:39,084][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:48:39,583][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:48:40,082][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:48:40,581][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:48:41,088][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:48:41,588][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:48:42,090][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:48:42,599][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:48:43,100][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:48:43,616][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:48:44,120][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:48:44,622][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:48:45,124][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:48:45,624][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:48:46,145][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:48:46,649][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10209 tokens.
[2025-11-13 02:48:47,368][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 02:48:48,143][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:48:48,144][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:48:48,146][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:48:49,060][__main__][INFO] - Iteration 313 took 52s (31.72% Gen, 66.55% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 24m 43s. Estimated total time: 44h 3m 50s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 7s, 500 more iterations: 7h 20m 38s.
[2025-11-13 02:48:49,062][__main__][INFO] - Starting iteration 313.
[2025-11-13 02:48:49,541][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:48:49,542][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:49:06,464][__main__][INFO] - Number of regex retries in iteration 313: 0
[2025-11-13 02:49:06,464][__main__][INFO] - agents played in iteration 313 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:49:07,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:49:07,355][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:49:07,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:49:07,402][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:49:07,403][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:49:07,403][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:49:08,123][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:49:08,579][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:49:09,086][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:49:09,588][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:49:10,100][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:49:10,600][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:49:11,097][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:49:11,608][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:49:12,107][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:49:12,624][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:49:13,127][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:49:13,628][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:49:14,125][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:49:14,627][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:49:15,131][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:49:15,630][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:49:16,133][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:49:16,639][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:49:17,138][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:49:17,640][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:49:18,145][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:49:18,649][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:49:19,152][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:49:19,652][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:49:20,151][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:49:20,650][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:49:21,152][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:49:21,652][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:49:22,155][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:49:22,653][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:49:23,156][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:49:23,654][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:49:24,157][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:49:24,654][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:49:25,149][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:49:25,646][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:49:26,145][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:49:26,644][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:49:27,144][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:49:27,643][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:49:28,146][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:49:28,645][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:49:29,143][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:49:29,643][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:49:30,142][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:49:30,650][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:49:31,149][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:49:31,651][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:49:32,151][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:49:32,649][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:49:33,149][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:49:33,645][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:49:34,143][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:49:34,646][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:49:35,144][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:49:35,643][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:49:36,146][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:49:36,649][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:49:37,150][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:49:37,650][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:49:38,150][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:49:38,656][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:49:39,155][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:49:39,655][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:49:40,154][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10202 tokens.
[2025-11-13 02:49:40,883][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 02:49:41,668][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:49:41,670][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:49:41,672][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:49:42,649][__main__][INFO] - Iteration 314 took 53s (31.86% Gen, 66.29% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 35m 25s. Estimated total time: 44h 15m 26s. Time estimates for 10 more iterations: 8m 51s, 100 more iterations: 1h 28m 30s, 500 more iterations: 7h 22m 34s.
[2025-11-13 02:49:42,651][__main__][INFO] - Starting iteration 314.
[2025-11-13 02:49:43,123][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:49:43,124][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:49:52,647][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:49:57,617][__main__][INFO] - Number of regex retries in iteration 314: 1
[2025-11-13 02:49:57,618][__main__][INFO] - agents played in iteration 314 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:49:58,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:49:58,464][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:49:58,488][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:49:58,510][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:49:58,510][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:49:58,511][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:49:59,196][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:49:59,655][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:50:00,157][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:50:00,659][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:50:01,158][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:50:01,656][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:50:02,161][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:50:02,663][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:50:03,159][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:50:03,661][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:50:04,162][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:50:04,663][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:50:05,162][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:50:05,661][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:50:06,163][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:50:06,663][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:50:07,164][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:50:07,670][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:50:08,168][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:50:08,670][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:50:09,168][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:50:09,684][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:50:10,186][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:50:10,688][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:50:11,186][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:50:11,690][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:50:12,189][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:50:12,691][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:50:13,190][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:50:13,690][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:50:14,191][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:50:14,694][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:50:15,193][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:50:15,694][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:50:16,196][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:50:16,700][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:50:17,202][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:50:17,706][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:50:18,209][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:50:18,714][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:50:19,217][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:50:19,718][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:50:20,218][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:50:20,735][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:50:21,239][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:50:21,743][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:50:22,245][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:50:22,747][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:50:23,252][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:50:23,750][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:50:24,249][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:50:24,758][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:50:25,257][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:50:25,756][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:50:26,257][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:50:26,754][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:50:27,259][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:50:27,759][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:50:28,257][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:50:28,763][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:50:29,264][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:50:29,764][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:50:30,266][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:50:30,764][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:50:31,268][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10231 tokens.
[2025-11-13 02:50:32,058][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 02:50:32,842][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:50:32,844][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:50:32,846][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:50:33,774][__main__][INFO] - Iteration 315 took 50s (28.62% Gen, 69.55% Train). Generation: 14s, Training: 35s. Estimated remaining time: 37h 31m 43s. Estimated total time: 42h 12m 35s. Time estimates for 10 more iterations: 8m 26s, 100 more iterations: 1h 24m 25s, 500 more iterations: 7h 2m 5s.
[2025-11-13 02:50:33,777][__main__][INFO] - Starting iteration 315.
[2025-11-13 02:50:34,273][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:50:34,273][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:50:38,326][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:50:50,797][__main__][INFO] - Number of regex retries in iteration 315: 1
[2025-11-13 02:50:50,798][__main__][INFO] - agents played in iteration 315 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:50:51,635][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:50:51,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:50:51,689][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:50:51,712][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:50:51,712][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:50:51,713][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:50:52,395][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:50:52,847][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:50:53,350][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:50:53,848][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:50:54,346][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:50:54,857][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:50:55,356][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:50:55,855][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:50:56,360][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:50:56,867][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:50:57,367][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:50:57,864][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:50:58,363][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:50:58,866][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:50:59,365][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:50:59,865][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:51:00,362][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:51:00,860][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:51:01,363][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:51:01,860][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:51:02,355][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:51:02,853][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:51:03,350][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:51:03,854][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:51:04,350][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:51:04,852][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:51:05,365][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:51:05,864][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:51:06,363][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:51:06,860][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:51:07,359][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:51:07,863][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:51:08,364][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:51:08,869][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:51:09,372][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:51:09,874][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:51:10,378][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:51:10,881][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:51:11,386][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:51:11,897][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:51:12,401][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:51:12,901][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:51:13,406][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:51:13,906][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:51:14,410][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:51:14,914][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:51:15,418][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:51:15,930][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:51:16,434][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:51:16,943][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:51:17,441][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:51:17,938][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:51:18,437][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:51:18,939][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:51:19,438][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:51:19,939][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:51:20,436][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:51:20,949][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:51:21,448][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:51:21,950][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:51:22,449][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:51:22,951][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:51:23,446][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:51:23,944][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:51:24,451][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10221 tokens.
[2025-11-13 02:51:25,195][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 02:51:25,971][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:51:25,973][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:51:25,974][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:51:26,997][__main__][INFO] - Iteration 316 took 52s (31.34% Gen, 66.72% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 14m 30s. Estimated total time: 43h 56m 15s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 52s, 500 more iterations: 7h 19m 22s.
[2025-11-13 02:51:26,999][__main__][INFO] - Starting iteration 316.
[2025-11-13 02:51:27,483][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:51:27,483][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:51:45,250][__main__][INFO] - Number of regex retries in iteration 316: 0
[2025-11-13 02:51:45,251][__main__][INFO] - agents played in iteration 316 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:51:46,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:51:46,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:51:46,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:51:46,106][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:51:46,107][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:51:46,108][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:51:46,795][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:51:47,259][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:51:47,763][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:51:48,259][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:51:48,760][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:51:49,255][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:51:49,765][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:51:50,266][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:51:50,765][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:51:51,280][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:51:51,784][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:51:52,284][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:51:52,788][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:51:53,290][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:51:53,799][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:51:54,298][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:51:54,800][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:51:55,301][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:51:55,926][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:51:56,449][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:51:56,945][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:51:57,446][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:51:57,949][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:51:58,447][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:51:58,958][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:51:59,460][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:51:59,957][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:52:00,457][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:52:00,962][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:52:01,470][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:52:01,970][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:52:02,471][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:52:02,978][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:52:03,480][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:52:03,982][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:52:04,481][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:52:04,982][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:52:05,491][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:52:05,991][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:52:06,492][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:52:06,992][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:52:07,493][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:52:07,994][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:52:08,494][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:52:08,993][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:52:09,495][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:52:09,991][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:52:10,489][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:52:10,987][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:52:11,492][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:52:11,993][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:52:12,490][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:52:12,988][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:52:13,489][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:52:13,986][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:52:14,483][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:52:14,979][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:52:15,475][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:52:15,976][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:52:16,476][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:52:16,970][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:52:17,475][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:52:17,971][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:52:18,468][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:52:18,975][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10266 tokens.
[2025-11-13 02:52:19,709][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 02:52:20,495][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:52:20,497][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:52:20,499][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:52:21,433][__main__][INFO] - Iteration 317 took 53s (32.93% Gen, 65.33% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 14m 51s. Estimated total time: 44h 57m 31s. Time estimates for 10 more iterations: 8m 59s, 100 more iterations: 1h 29m 55s, 500 more iterations: 7h 29m 35s.
[2025-11-13 02:52:21,435][__main__][INFO] - Starting iteration 317.
[2025-11-13 02:52:21,927][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:52:21,928][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:52:38,652][__main__][INFO] - Number of regex retries in iteration 317: 0
[2025-11-13 02:52:38,652][__main__][INFO] - agents played in iteration 317 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:52:39,476][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:52:39,503][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:52:39,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:52:39,553][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:52:39,553][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:52:39,554][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:52:40,235][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:52:40,690][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:52:41,194][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:52:41,692][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:52:42,197][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:52:42,693][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:52:43,191][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:52:43,691][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:52:44,190][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:52:44,695][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:52:45,198][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:52:45,701][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:52:46,204][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:52:46,707][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:52:47,209][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:52:47,714][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:52:48,213][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:52:48,711][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:52:49,210][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:52:49,710][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:52:50,215][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:52:50,713][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:52:51,212][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:52:51,716][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:52:52,217][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:52:52,715][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:52:53,218][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:52:53,722][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:52:54,220][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:52:54,721][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:52:55,223][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:52:55,729][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:52:56,233][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:52:56,735][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:52:57,239][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:52:57,740][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:52:58,248][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:52:58,749][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:52:59,263][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:52:59,763][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:53:00,261][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:53:00,775][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:53:01,278][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:53:01,779][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:53:02,277][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:53:02,775][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:53:03,281][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:53:03,778][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:53:04,275][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:53:04,776][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:53:05,274][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:53:05,772][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:53:06,271][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:53:06,772][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:53:07,285][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:53:07,784][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:53:08,282][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:53:08,785][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:53:09,282][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:53:09,785][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:53:10,285][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:53:10,787][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:53:11,290][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:53:11,791][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:53:12,292][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10242 tokens.
[2025-11-13 02:53:13,014][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 02:53:13,815][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:53:13,817][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:53:13,820][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:53:14,981][__main__][INFO] - Iteration 318 took 53s (31.52% Gen, 66.29% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 29m 10s. Estimated total time: 44h 12m 43s. Time estimates for 10 more iterations: 8m 50s, 100 more iterations: 1h 28m 25s, 500 more iterations: 7h 22m 7s.
[2025-11-13 02:53:14,983][__main__][INFO] - Starting iteration 318.
[2025-11-13 02:53:15,454][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:53:15,455][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:53:33,924][__main__][INFO] - Number of regex retries in iteration 318: 0
[2025-11-13 02:53:33,925][__main__][INFO] - agents played in iteration 318 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:53:34,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:53:34,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:53:34,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:53:34,829][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:53:34,830][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:53:34,831][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:53:35,499][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:53:35,951][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:53:36,464][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:53:37,003][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:53:37,506][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:53:38,004][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:53:38,499][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:53:39,008][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:53:39,509][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:53:40,014][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:53:40,518][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:53:41,018][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:53:41,519][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:53:42,026][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:53:42,527][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:53:43,034][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:53:43,535][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:53:44,035][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:53:44,535][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:53:45,036][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:53:45,539][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:53:46,041][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:53:46,542][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:53:47,041][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:53:47,540][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:53:48,042][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:53:48,547][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:53:49,049][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:53:49,550][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:53:50,050][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:53:50,552][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:53:51,053][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:53:51,557][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:53:52,054][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:53:52,554][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:53:53,052][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:53:53,559][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:53:54,057][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:53:54,557][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:53:55,067][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:53:55,566][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:53:56,067][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:53:56,566][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:53:57,063][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:53:57,562][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:53:58,062][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:53:58,560][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:53:59,063][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:53:59,564][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:54:00,078][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:54:00,581][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:54:01,078][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:54:01,593][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:54:02,097][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:54:02,598][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:54:03,098][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:54:03,598][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:54:04,102][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:54:04,599][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:54:05,098][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:54:05,598][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:54:06,101][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:54:06,605][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:54:07,101][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:54:07,601][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10227 tokens.
[2025-11-13 02:54:08,398][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.13%, ΔTime: 00:00:32
[2025-11-13 02:54:09,179][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:54:09,180][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:54:09,183][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:54:10,375][__main__][INFO] - Iteration 319 took 54s (33.63% Gen, 64.20% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 1m 34s. Estimated total time: 45h 46m 3s. Time estimates for 10 more iterations: 9m 9s, 100 more iterations: 1h 31m 32s, 500 more iterations: 7h 37m 40s.
[2025-11-13 02:54:10,377][__main__][INFO] - Starting iteration 319.
[2025-11-13 02:54:10,866][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:54:10,867][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:54:28,482][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:54:29,403][__main__][INFO] - Number of regex retries in iteration 319: 1
[2025-11-13 02:54:29,404][__main__][INFO] - agents played in iteration 319 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:54:30,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:54:30,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:54:30,265][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:54:30,291][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:54:30,291][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:54:30,292][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:54:30,961][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:54:31,412][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:54:31,915][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:54:32,412][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:54:32,913][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:54:33,413][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:54:33,909][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:54:34,409][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:54:34,904][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:54:35,398][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:54:35,902][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:54:36,404][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:54:36,907][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:54:37,407][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:54:37,905][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:54:38,400][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:54:38,897][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:54:39,393][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:54:39,888][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:54:40,386][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:54:40,897][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:54:41,393][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:54:41,895][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:54:42,394][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:54:42,896][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:54:43,425][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:54:43,930][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:54:44,437][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:54:44,939][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:54:45,443][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:54:45,944][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:54:46,447][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:54:46,949][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:54:47,451][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:54:47,951][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:54:48,455][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:54:48,958][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:54:49,462][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:54:49,964][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:54:50,463][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:54:50,967][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:54:51,484][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:54:51,983][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:54:52,485][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:54:52,985][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:54:53,484][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:54:54,002][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:54:54,500][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:54:55,006][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:54:55,506][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:54:56,014][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:54:56,517][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:54:57,019][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:54:57,519][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:54:58,021][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:54:58,524][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:54:59,027][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:54:59,527][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:55:00,031][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:55:00,533][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:55:01,033][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:55:01,533][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:55:02,034][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:55:02,532][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:55:03,029][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10261 tokens.
[2025-11-13 02:55:03,795][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 02:55:04,577][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:55:04,578][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:55:04,580][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:55:05,528][__main__][INFO] - Iteration 320 took 54s (33.91% Gen, 64.35% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 47m 42s. Estimated total time: 45h 33m 5s. Time estimates for 10 more iterations: 9m 6s, 100 more iterations: 1h 31m 6s, 500 more iterations: 7h 35m 30s.
[2025-11-13 02:55:05,530][__main__][INFO] - Starting iteration 320.
[2025-11-13 02:55:06,001][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 02:55:06,002][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:55:09,820][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:55:21,892][__main__][INFO] - Number of regex retries in iteration 320: 1
[2025-11-13 02:55:21,893][__main__][INFO] - agents played in iteration 320 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:55:22,682][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:55:22,712][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:55:22,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:55:22,762][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:55:22,763][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:55:22,763][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:55:23,443][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:55:23,901][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:55:24,404][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:55:24,906][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:55:25,404][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:55:25,904][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:55:26,402][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:55:26,903][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:55:27,404][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:55:27,903][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:55:28,401][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:55:28,900][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:55:29,412][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:55:29,910][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:55:30,411][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:55:30,913][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:55:31,410][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:55:31,909][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:55:32,407][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:55:32,907][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:55:33,417][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:55:33,917][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:55:34,418][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:55:34,920][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:55:35,422][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:55:35,928][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:55:36,426][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:55:36,926][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:55:37,435][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:55:37,935][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:55:38,434][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:55:38,941][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:55:39,439][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:55:39,943][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:55:40,443][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:55:40,942][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:55:41,445][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:55:41,945][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:55:42,446][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:55:42,953][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:55:43,456][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:55:43,954][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:55:44,453][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:55:44,955][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:55:45,456][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:55:45,958][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:55:46,459][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:55:46,964][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:55:47,462][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:55:47,960][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:55:48,458][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:55:48,956][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:55:49,456][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:55:49,956][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:55:50,454][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:55:50,957][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:55:51,459][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:55:51,960][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:55:52,459][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:55:52,963][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:55:53,464][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:55:53,966][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:55:54,468][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:55:54,966][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:55:55,468][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10256 tokens.
[2025-11-13 02:55:56,270][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 02:55:57,054][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:55:57,056][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:55:57,058][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:55:58,943][__main__][INFO] - Iteration 321 took 52s (30.02% Gen, 66.42% Train). Generation: 15s, Training: 35s. Estimated remaining time: 39h 20m 48s. Estimated total time: 44h 7m 6s. Time estimates for 10 more iterations: 8m 49s, 100 more iterations: 1h 28m 14s, 500 more iterations: 7h 21m 11s.
[2025-11-13 02:55:58,945][__main__][INFO] - Starting iteration 321.
[2025-11-13 02:55:59,427][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 02:55:59,428][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:56:16,497][__main__][INFO] - Number of regex retries in iteration 321: 0
[2025-11-13 02:56:16,498][__main__][INFO] - agents played in iteration 321 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:56:17,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:56:17,310][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:56:17,336][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:56:17,359][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:56:17,360][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:56:17,361][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:56:18,025][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:56:18,500][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:56:18,999][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:56:19,494][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:56:19,995][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:56:20,490][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:56:20,992][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:56:21,489][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:56:21,985][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:56:22,500][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:56:23,000][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:56:23,498][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:56:23,995][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:56:24,491][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:56:25,005][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:56:25,505][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:56:26,000][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:56:26,513][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:56:27,012][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:56:27,512][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:56:28,010][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:56:28,511][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:56:29,023][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:56:29,521][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:56:30,024][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:56:30,528][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:56:31,031][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:56:31,538][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:56:32,039][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:56:32,539][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:56:33,045][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:56:33,545][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:56:34,046][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:56:34,548][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:56:35,049][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:56:35,554][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:56:36,054][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:56:36,554][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:56:37,057][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:56:37,554][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:56:38,056][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:56:38,555][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:56:39,053][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:56:39,554][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:56:40,052][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:56:40,553][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:56:41,056][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:56:41,553][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:56:42,053][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:56:42,550][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:56:43,047][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:56:43,548][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:56:44,052][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:56:44,562][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:56:45,073][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:56:45,582][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:56:46,109][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:56:46,619][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:56:47,118][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:56:47,625][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:56:48,125][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:56:48,627][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:56:49,125][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:56:49,626][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:56:50,135][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10327 tokens.
[2025-11-13 02:56:50,955][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:32
[2025-11-13 02:56:51,728][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:56:51,730][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:56:51,734][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:56:52,944][__main__][INFO] - Iteration 322 took 53s (31.89% Gen, 65.84% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 48m 40s. Estimated total time: 44h 35m 52s. Time estimates for 10 more iterations: 8m 55s, 100 more iterations: 1h 29m 11s, 500 more iterations: 7h 25m 58s.
[2025-11-13 02:56:52,946][__main__][INFO] - Starting iteration 322.
[2025-11-13 02:56:53,439][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 02:56:53,440][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:57:11,370][__main__][INFO] - Number of regex retries in iteration 322: 0
[2025-11-13 02:57:11,371][__main__][INFO] - agents played in iteration 322 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:57:12,159][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:57:12,188][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:57:12,216][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:57:12,241][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:57:12,242][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:57:12,242][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:57:12,912][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:57:13,366][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:57:13,879][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:57:14,377][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:57:14,873][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:57:15,374][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:57:15,870][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:57:16,371][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:57:16,868][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:57:17,368][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:57:17,867][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:57:18,364][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:57:18,863][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:57:19,366][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:57:19,867][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:57:20,370][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:57:20,870][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:57:21,370][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:57:21,870][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:57:22,367][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:57:22,870][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:57:23,372][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:57:23,873][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:57:24,380][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:57:24,880][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:57:25,380][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:57:25,879][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:57:26,378][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:57:26,885][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:57:27,382][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:57:27,879][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:57:28,391][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:57:28,892][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:57:29,397][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:57:29,896][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:57:30,395][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:57:30,894][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:57:31,392][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:57:31,890][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:57:32,389][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:57:32,888][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:57:33,406][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:57:33,905][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:57:34,403][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:57:34,909][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:57:35,410][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:57:35,920][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:57:36,418][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:57:36,916][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:57:37,420][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:57:37,921][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:57:38,425][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:57:38,928][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:57:39,430][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:57:39,931][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:57:40,430][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:57:40,929][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:57:41,430][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:57:41,928][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:57:42,428][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:57:42,926][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:57:43,426][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:57:43,935][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:57:44,433][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:57:44,937][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10230 tokens.
[2025-11-13 02:57:45,795][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 02:57:46,611][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:57:46,612][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:57:46,614][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:57:47,651][__main__][INFO] - Iteration 323 took 54s (33.07% Gen, 65.01% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 22m 30s. Estimated total time: 45h 10m 36s. Time estimates for 10 more iterations: 9m 2s, 100 more iterations: 1h 30m 21s, 500 more iterations: 7h 31m 46s.
[2025-11-13 02:57:47,653][__main__][INFO] - Starting iteration 323.
[2025-11-13 02:57:48,141][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 02:57:48,142][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:58:04,707][__main__][INFO] - Number of regex retries in iteration 323: 0
[2025-11-13 02:58:04,708][__main__][INFO] - agents played in iteration 323 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:58:05,482][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:58:05,507][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:58:05,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:58:05,555][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:58:05,556][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:58:05,557][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:58:06,238][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:58:06,696][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:58:07,197][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:58:07,708][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:58:08,210][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:58:08,706][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:58:09,224][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:58:09,724][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:58:10,222][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:58:10,719][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:58:11,218][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:58:11,731][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:58:12,230][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:58:12,729][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:58:13,234][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:58:13,731][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:58:14,241][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:58:14,743][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:58:15,240][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:58:15,741][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:58:16,240][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:58:16,738][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:58:17,241][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:58:17,742][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:58:18,248][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:58:18,748][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:58:19,252][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:58:19,759][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:58:20,257][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:58:20,756][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:58:21,257][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:58:21,756][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:58:22,263][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:58:22,765][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:58:23,264][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:58:23,763][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:58:24,264][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:58:24,764][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:58:25,262][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:58:25,762][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:58:26,268][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:58:26,767][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:58:27,270][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:58:27,770][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:58:28,269][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:58:28,770][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:58:29,269][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:58:29,767][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:58:30,265][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:58:30,763][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:58:31,264][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:58:31,760][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:58:32,256][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:58:32,755][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:58:33,252][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:58:33,750][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:58:34,248][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:58:34,746][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:58:35,249][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:58:35,746][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:58:36,247][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:58:36,744][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:58:37,241][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:58:37,739][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:58:38,241][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10197 tokens.
[2025-11-13 02:58:38,975][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 02:58:39,759][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:58:39,761][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:58:39,762][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:58:41,118][__main__][INFO] - Iteration 324 took 52s (31.27% Gen, 66.17% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 19m 53s. Estimated total time: 44h 8m 53s. Time estimates for 10 more iterations: 8m 49s, 100 more iterations: 1h 28m 17s, 500 more iterations: 7h 21m 28s.
[2025-11-13 02:58:41,120][__main__][INFO] - Starting iteration 324.
[2025-11-13 02:58:41,628][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 02:58:41,629][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:59:00,628][__main__][INFO] - Number of regex retries in iteration 324: 0
[2025-11-13 02:59:00,628][__main__][INFO] - agents played in iteration 324 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:59:01,403][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:59:01,430][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:59:01,455][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:59:01,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:59:01,478][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:59:01,479][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:59:02,157][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:59:02,611][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:59:03,111][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:59:03,611][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:59:04,118][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:59:04,617][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:59:05,119][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:59:05,632][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:59:06,127][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 02:59:06,637][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 02:59:07,140][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 02:59:07,642][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 02:59:08,146][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 02:59:08,648][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 02:59:09,151][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 02:59:09,650][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 02:59:10,150][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 02:59:10,660][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 02:59:11,156][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 02:59:11,657][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 02:59:12,153][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 02:59:12,650][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 02:59:13,159][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 02:59:13,661][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 02:59:14,161][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 02:59:14,661][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 02:59:15,162][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 02:59:15,669][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 02:59:16,169][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 02:59:16,674][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 02:59:17,177][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 02:59:17,678][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 02:59:18,184][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 02:59:18,697][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 02:59:19,200][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 02:59:19,709][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 02:59:20,216][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 02:59:20,719][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 02:59:21,234][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 02:59:21,739][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 02:59:22,243][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 02:59:22,742][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 02:59:23,246][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 02:59:23,758][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 02:59:24,261][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 02:59:24,757][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 02:59:25,257][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 02:59:25,754][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 02:59:26,250][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 02:59:26,779][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 02:59:27,281][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 02:59:27,783][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 02:59:28,289][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 02:59:28,790][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 02:59:29,292][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 02:59:29,799][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 02:59:30,307][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 02:59:30,803][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 02:59:31,305][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 02:59:31,806][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 02:59:32,305][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 02:59:32,812][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 02:59:33,316][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 02:59:33,818][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 02:59:34,326][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10363 tokens.
[2025-11-13 02:59:35,105][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.36%, ΔTime: 00:00:32
[2025-11-13 02:59:35,883][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 02:59:35,885][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 02:59:35,887][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 02:59:36,937][__main__][INFO] - Iteration 325 took 55s (34.35% Gen, 63.75% Train). Generation: 18s, Training: 35s. Estimated remaining time: 41h 15m 32s. Estimated total time: 46h 5m 28s. Time estimates for 10 more iterations: 9m 13s, 100 more iterations: 1h 32m 10s, 500 more iterations: 7h 40m 54s.
[2025-11-13 02:59:36,939][__main__][INFO] - Starting iteration 325.
[2025-11-13 02:59:37,406][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 02:59:37,407][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 02:59:42,132][mllm.models.large_language_model_local][WARNING] - Response Proposition: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 02:59:54,337][__main__][INFO] - Number of regex retries in iteration 325: 1
[2025-11-13 02:59:54,338][__main__][INFO] - agents played in iteration 325 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 02:59:55,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:59:55,159][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:59:55,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:59:55,209][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 02:59:55,209][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 02:59:55,210][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 02:59:56,012][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 02:59:56,476][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 02:59:56,979][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 02:59:57,484][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 02:59:57,982][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 02:59:58,482][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 02:59:58,981][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 02:59:59,484][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 02:59:59,982][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:00:00,484][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:00:00,989][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:00:01,489][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:00:01,987][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:00:02,488][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:00:02,990][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:00:03,486][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:00:03,989][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:00:04,487][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:00:04,988][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:00:05,488][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:00:05,988][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:00:06,487][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:00:06,986][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:00:07,484][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:00:07,985][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:00:08,489][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:00:08,991][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:00:09,502][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:00:10,007][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:00:10,509][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:00:11,014][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:00:11,519][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:00:12,028][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:00:12,529][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:00:13,031][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:00:13,532][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:00:14,031][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:00:14,536][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:00:15,036][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:00:15,536][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:00:16,049][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:00:16,551][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:00:17,060][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:00:17,558][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:00:18,062][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:00:18,578][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:00:19,080][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:00:19,581][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:00:20,078][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:00:20,574][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:00:21,083][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:00:21,578][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:00:22,078][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:00:22,575][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:00:23,074][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:00:23,577][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:00:24,072][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:00:24,570][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:00:25,082][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:00:25,582][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:00:26,081][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:00:26,582][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:00:27,083][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:00:27,590][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:00:28,091][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10164 tokens.
[2025-11-13 03:00:28,832][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 03:00:29,628][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:00:29,630][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:00:29,632][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:00:30,600][__main__][INFO] - Iteration 326 took 53s (31.83% Gen, 66.35% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 28m 53s. Estimated total time: 44h 19m 42s. Time estimates for 10 more iterations: 8m 51s, 100 more iterations: 1h 28m 39s, 500 more iterations: 7h 23m 17s.
[2025-11-13 03:00:30,603][__main__][INFO] - Starting iteration 326.
[2025-11-13 03:00:31,074][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 03:00:31,075][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:00:48,173][__main__][INFO] - Number of regex retries in iteration 326: 0
[2025-11-13 03:00:48,173][__main__][INFO] - agents played in iteration 326 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:00:49,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:00:49,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:00:49,068][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:00:49,090][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:00:49,091][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:00:49,092][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:00:49,848][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:00:50,312][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:00:50,815][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:00:51,319][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:00:51,820][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:00:52,324][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:00:52,823][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:00:53,319][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:00:53,816][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:00:54,318][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:00:54,819][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:00:55,320][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:00:55,819][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:00:56,317][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:00:56,817][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:00:57,317][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:00:57,817][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:00:58,321][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:00:58,824][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:00:59,328][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:00:59,831][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:01:00,331][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:01:00,830][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:01:01,341][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:01:01,840][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:01:02,352][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:01:02,852][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:01:03,353][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:01:03,861][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:01:04,366][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:01:04,877][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:01:05,375][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:01:05,874][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:01:06,387][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:01:06,886][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:01:07,389][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:01:07,889][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:01:08,392][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:01:08,899][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:01:09,401][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:01:09,905][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:01:10,410][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:01:10,912][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:01:11,414][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:01:11,918][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:01:12,421][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:01:12,921][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:01:13,422][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:01:13,923][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:01:14,427][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:01:14,927][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:01:15,424][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:01:15,922][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:01:16,422][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:01:16,920][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:01:17,421][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:01:17,919][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:01:18,423][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:01:18,923][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:01:19,422][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:01:19,921][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:01:20,420][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:01:20,926][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:01:21,425][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:01:21,946][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10318 tokens.
[2025-11-13 03:01:22,723][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.29%, Current % of VRAM taken: 60.55%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 03:01:23,532][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:01:23,534][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:01:23,535][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:01:24,742][__main__][INFO] - Iteration 327 took 53s (31.86% Gen, 65.89% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 51m 44s. Estimated total time: 44h 43m 27s. Time estimates for 10 more iterations: 8m 56s, 100 more iterations: 1h 29m 26s, 500 more iterations: 7h 27m 14s.
[2025-11-13 03:01:24,744][__main__][INFO] - Starting iteration 327.
[2025-11-13 03:01:25,213][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 03:01:25,215][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:01:40,860][__main__][INFO] - Number of regex retries in iteration 327: 0
[2025-11-13 03:01:40,861][__main__][INFO] - agents played in iteration 327 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:01:41,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:01:41,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:01:41,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:01:41,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:01:41,785][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:01:41,786][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:01:42,513][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:01:42,966][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:01:43,468][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:01:43,969][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:01:44,469][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:01:44,974][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:01:45,474][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:01:45,975][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:01:46,483][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:01:46,995][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:01:47,502][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:01:48,011][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:01:48,518][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:01:49,016][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:01:49,514][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:01:50,012][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:01:50,510][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:01:51,014][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:01:51,513][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:01:52,014][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:01:52,514][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:01:53,019][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:01:53,541][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:01:54,053][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:01:54,558][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:01:55,071][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:01:55,576][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:01:56,080][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:01:56,580][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:01:57,087][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:01:57,594][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:01:58,097][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:01:58,600][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:01:59,108][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:01:59,608][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:02:00,107][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:02:00,605][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:02:01,105][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:02:01,608][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:02:02,109][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:02:02,609][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:02:03,118][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:02:03,618][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:02:04,118][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:02:04,619][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:02:05,118][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:02:05,622][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:02:06,122][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:02:06,622][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:02:07,123][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:02:07,624][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:02:08,126][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:02:08,627][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:02:09,127][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:02:09,628][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:02:10,127][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:02:10,629][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:02:11,132][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:02:11,631][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:02:12,133][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:02:12,635][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:02:13,132][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:02:13,636][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:02:14,134][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:02:14,637][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10189 tokens.
[2025-11-13 03:02:15,427][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 03:02:16,218][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:02:16,220][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:02:16,221][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:02:17,166][__main__][INFO] - Iteration 328 took 51s (30.11% Gen, 68.06% Train). Generation: 15s, Training: 35s. Estimated remaining time: 38h 25m 2s. Estimated total time: 43h 17m 38s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 35s, 500 more iterations: 7h 12m 56s.
[2025-11-13 03:02:17,168][__main__][INFO] - Starting iteration 328.
[2025-11-13 03:02:17,641][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 03:02:17,642][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:02:36,242][__main__][INFO] - Number of regex retries in iteration 328: 0
[2025-11-13 03:02:36,243][__main__][INFO] - agents played in iteration 328 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:02:37,135][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:02:37,157][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:02:37,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:02:37,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:02:37,203][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:02:37,204][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:02:37,924][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:02:38,392][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:02:38,900][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:02:39,399][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:02:39,899][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:02:40,397][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:02:40,900][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:02:41,397][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:02:41,898][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:02:42,400][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:02:42,897][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:02:43,392][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:02:43,891][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:02:44,391][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:02:44,896][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:02:45,391][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:02:45,887][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:02:46,385][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:02:46,884][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:02:47,389][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:02:47,887][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:02:48,387][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:02:48,897][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:02:49,396][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:02:49,902][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:02:50,404][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:02:50,907][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:02:51,411][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:02:51,912][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:02:52,415][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:02:52,914][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:02:53,416][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:02:53,914][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:02:54,424][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:02:54,924][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:02:55,426][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:02:55,926][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:02:56,428][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:02:56,937][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:02:57,447][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:02:57,964][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:02:58,477][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:02:58,991][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:02:59,493][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:02:59,996][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:03:00,503][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:03:01,001][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:03:01,501][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:03:01,998][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:03:02,499][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:03:02,999][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:03:03,502][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:03:04,005][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:03:04,507][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:03:05,008][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:03:05,515][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:03:06,014][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:03:06,514][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:03:07,013][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:03:07,512][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:03:08,013][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:03:08,514][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:03:09,015][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:03:09,516][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:03:10,014][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10276 tokens.
[2025-11-13 03:03:10,754][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 03:03:11,532][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:03:11,533][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:03:11,535][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:03:12,585][__main__][INFO] - Iteration 329 took 54s (33.85% Gen, 64.23% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 53m 41s. Estimated total time: 45h 47m 12s. Time estimates for 10 more iterations: 9m 9s, 100 more iterations: 1h 31m 34s, 500 more iterations: 7h 37m 52s.
[2025-11-13 03:03:12,587][__main__][INFO] - Starting iteration 329.
[2025-11-13 03:03:13,059][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 03:03:13,060][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:03:22,991][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:03:31,493][__main__][INFO] - Number of regex retries in iteration 329: 1
[2025-11-13 03:03:31,493][__main__][INFO] - agents played in iteration 329 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:03:32,338][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:03:32,362][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:03:32,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:03:32,407][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:03:32,408][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:03:32,409][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:03:33,122][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:03:33,574][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:03:34,081][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:03:34,584][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:03:35,086][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:03:35,582][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:03:36,091][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:03:36,589][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:03:37,102][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:03:37,604][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:03:38,100][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:03:38,615][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:03:39,113][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:03:39,611][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:03:40,109][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:03:40,606][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:03:41,114][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:03:41,614][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:03:42,113][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:03:42,616][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:03:43,116][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:03:43,615][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:03:44,115][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:03:44,618][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:03:45,128][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:03:45,631][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:03:46,138][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:03:46,640][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:03:47,144][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:03:47,647][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:03:48,149][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:03:48,646][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:03:49,145][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:03:49,647][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:03:50,149][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:03:50,649][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:03:51,152][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:03:51,651][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:03:52,152][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:03:52,649][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:03:53,148][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:03:53,644][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:03:54,141][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:03:54,638][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:03:55,137][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:03:55,637][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:03:56,136][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:03:56,639][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:03:57,137][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:03:57,638][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:03:58,140][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:03:58,638][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:03:59,137][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:03:59,643][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:04:00,142][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:04:00,643][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:04:01,142][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:04:01,641][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:04:02,145][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:04:02,649][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:04:03,150][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:04:03,649][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:04:04,148][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:04:04,644][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:04:05,139][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10294 tokens.
[2025-11-13 03:04:05,864][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 03:04:06,643][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:04:06,645][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:04:06,646][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:04:07,585][__main__][INFO] - Iteration 330 took 54s (33.81% Gen, 64.47% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 31m 54s. Estimated total time: 45h 26m 20s. Time estimates for 10 more iterations: 9m 5s, 100 more iterations: 1h 30m 52s, 500 more iterations: 7h 34m 23s.
[2025-11-13 03:04:07,587][__main__][INFO] - Starting iteration 330.
[2025-11-13 03:04:08,057][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 03:04:08,057][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:04:23,825][__main__][INFO] - Number of regex retries in iteration 330: 0
[2025-11-13 03:04:23,825][__main__][INFO] - agents played in iteration 330 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:04:24,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:04:24,646][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:04:24,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:04:24,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:04:24,693][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:04:24,693][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:04:25,418][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:04:25,874][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:04:26,376][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:04:26,875][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:04:27,375][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:04:27,873][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:04:28,372][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:04:28,871][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:04:29,369][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:04:29,874][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:04:30,372][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:04:30,872][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:04:31,370][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:04:31,867][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:04:32,365][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:04:32,860][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:04:33,362][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:04:33,862][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:04:34,364][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:04:34,863][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:04:35,360][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:04:35,862][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:04:36,363][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:04:36,865][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:04:37,369][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:04:37,876][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:04:38,377][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:04:38,879][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:04:39,384][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:04:39,881][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:04:40,381][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:04:40,880][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:04:41,380][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:04:41,885][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:04:42,383][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:04:42,882][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:04:43,389][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:04:43,888][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:04:44,388][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:04:44,884][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:04:45,380][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:04:45,882][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:04:46,376][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:04:46,872][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:04:47,369][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:04:47,868][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:04:48,367][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:04:48,866][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:04:49,361][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:04:49,861][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:04:50,354][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:04:50,856][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:04:51,357][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:04:51,855][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:04:52,355][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:04:52,852][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:04:53,352][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:04:53,853][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:04:54,351][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:04:54,854][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:04:55,354][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:04:55,854][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:04:56,355][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:04:56,855][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:04:57,356][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10263 tokens.
[2025-11-13 03:04:58,127][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:32
[2025-11-13 03:04:58,927][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:04:58,929][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:04:58,930][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:05:01,541][__main__][INFO] - Iteration 331 took 53s (29.48% Gen, 65.64% Train). Generation: 15s, Training: 35s. Estimated remaining time: 39h 38m 54s. Estimated total time: 44h 34m 14s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 8s, 500 more iterations: 7h 25m 42s.
[2025-11-13 03:05:01,543][__main__][INFO] - Starting iteration 331.
[2025-11-13 03:05:02,035][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:05:02,035][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:05:14,074][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:05:20,671][__main__][INFO] - Number of regex retries in iteration 331: 1
[2025-11-13 03:05:20,671][__main__][INFO] - agents played in iteration 331 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:05:21,465][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:05:21,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:05:21,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:05:21,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:05:21,542][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:05:21,543][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:05:22,263][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:05:22,723][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:05:23,228][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:05:23,749][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:05:24,248][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:05:24,750][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:05:25,249][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:05:25,748][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:05:26,256][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:05:26,753][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:05:27,256][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:05:27,753][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:05:28,249][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:05:28,754][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:05:29,253][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:05:29,755][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:05:30,261][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:05:30,758][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:05:31,254][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:05:31,762][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:05:32,261][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:05:32,759][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:05:33,259][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:05:33,758][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:05:34,258][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:05:34,757][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:05:35,259][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:05:35,758][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:05:36,256][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:05:36,762][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:05:37,264][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:05:37,763][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:05:38,265][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:05:38,764][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:05:39,264][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:05:39,762][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:05:40,261][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:05:40,761][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:05:41,261][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:05:41,756][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:05:42,261][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:05:42,759][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:05:43,260][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:05:43,758][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:05:44,256][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:05:44,757][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:05:45,253][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:05:45,750][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:05:46,249][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:05:46,748][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:05:47,251][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:05:47,752][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:05:48,258][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:05:48,764][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:05:49,264][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:05:49,764][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:05:50,271][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:05:50,771][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:05:51,272][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:05:51,777][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:05:52,278][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:05:52,781][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:05:53,284][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:05:53,784][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:05:54,290][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10298 tokens.
[2025-11-13 03:05:55,023][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 03:05:55,807][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:05:55,808][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:05:55,810][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:05:56,829][__main__][INFO] - Iteration 332 took 54s (34.01% Gen, 64.13% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 43m 28s. Estimated total time: 45h 39m 43s. Time estimates for 10 more iterations: 9m 7s, 100 more iterations: 1h 31m 19s, 500 more iterations: 7h 36m 37s.
[2025-11-13 03:05:56,831][__main__][INFO] - Starting iteration 332.
[2025-11-13 03:05:57,305][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:05:57,306][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:06:15,986][__main__][INFO] - Number of regex retries in iteration 332: 0
[2025-11-13 03:06:15,987][__main__][INFO] - agents played in iteration 332 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:06:16,791][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:06:16,814][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:06:16,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:06:16,858][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:06:16,858][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:06:16,859][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:06:17,565][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:06:18,026][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:06:18,531][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:06:19,047][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:06:19,556][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:06:20,054][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:06:20,560][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:06:21,059][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:06:21,562][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:06:22,056][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:06:22,556][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:06:23,067][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:06:23,566][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:06:24,067][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:06:24,562][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:06:25,062][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:06:25,562][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:06:26,057][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:06:26,557][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:06:27,068][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:06:27,565][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:06:28,065][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:06:28,569][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:06:29,068][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:06:29,575][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:06:30,076][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:06:30,572][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:06:31,073][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:06:31,575][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:06:32,074][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:06:32,571][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:06:33,072][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:06:33,580][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:06:34,080][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:06:34,580][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:06:35,080][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:06:35,581][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:06:36,077][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:06:36,574][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:06:37,072][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:06:37,579][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:06:38,075][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:06:38,574][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:06:39,068][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:06:39,565][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:06:40,067][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:06:40,563][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:06:41,058][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:06:41,555][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:06:42,054][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:06:42,560][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:06:43,058][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:06:43,557][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:06:44,073][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:06:44,572][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:06:45,075][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:06:45,579][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:06:46,081][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:06:46,589][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:06:47,098][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:06:47,621][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:06:48,126][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:06:48,632][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:06:49,139][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:06:49,639][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10339 tokens.
[2025-11-13 03:06:50,361][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 03:06:51,147][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:06:51,148][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:06:51,150][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:06:52,083][__main__][INFO] - Iteration 333 took 54s (34.10% Gen, 64.19% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 41m 44s. Estimated total time: 45h 38m 54s. Time estimates for 10 more iterations: 9m 7s, 100 more iterations: 1h 31m 17s, 500 more iterations: 7h 36m 29s.
[2025-11-13 03:06:52,085][__main__][INFO] - Starting iteration 333.
[2025-11-13 03:06:52,568][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:06:52,568][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:07:09,187][__main__][INFO] - Number of regex retries in iteration 333: 0
[2025-11-13 03:07:09,188][__main__][INFO] - agents played in iteration 333 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:07:10,059][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:07:10,087][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:07:10,113][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:07:10,136][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:07:10,136][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:07:10,137][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:07:10,826][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:07:11,281][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:07:11,794][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:07:12,295][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:07:12,795][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:07:13,298][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:07:13,796][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:07:14,294][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:07:14,795][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:07:15,290][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:07:15,791][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:07:16,286][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:07:16,782][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:07:17,281][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:07:17,776][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:07:18,276][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:07:18,775][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:07:19,273][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:07:19,779][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:07:20,281][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:07:20,783][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:07:21,283][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:07:21,784][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:07:22,288][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:07:22,784][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:07:23,283][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:07:23,783][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:07:24,281][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:07:24,780][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:07:25,275][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:07:25,773][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:07:26,278][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:07:26,776][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:07:27,275][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:07:27,775][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:07:28,271][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:07:28,768][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:07:29,265][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:07:29,762][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:07:30,272][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:07:30,769][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:07:31,265][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:07:31,766][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:07:32,265][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:07:32,760][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:07:33,255][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:07:33,750][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:07:34,248][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:07:34,747][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:07:35,243][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:07:35,748][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:07:36,244][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:07:36,742][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:07:37,244][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:07:37,740][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:07:38,241][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:07:38,740][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:07:39,238][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:07:39,750][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:07:40,252][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:07:40,766][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:07:41,261][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:07:41,761][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:07:42,275][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:07:42,776][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10198 tokens.
[2025-11-13 03:07:43,557][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:32
[2025-11-13 03:07:44,314][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:07:44,316][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:07:44,318][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:07:45,279][__main__][INFO] - Iteration 334 took 52s (31.53% Gen, 66.65% Train). Generation: 16s, Training: 35s. Estimated remaining time: 38h 57m 30s. Estimated total time: 43h 55m 34s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 51s, 500 more iterations: 7h 19m 15s.
[2025-11-13 03:07:45,281][__main__][INFO] - Starting iteration 334.
[2025-11-13 03:07:45,750][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:07:45,751][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:08:03,689][__main__][INFO] - Number of regex retries in iteration 334: 0
[2025-11-13 03:08:03,690][__main__][INFO] - agents played in iteration 334 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:08:04,537][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:08:04,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:08:04,589][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:08:04,611][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:08:04,612][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:08:04,613][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:08:05,289][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:08:05,743][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:08:06,257][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:08:06,754][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:08:07,251][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:08:07,754][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:08:08,250][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:08:08,767][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:08:09,264][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:08:09,759][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:08:10,256][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:08:10,752][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:08:11,252][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:08:11,749][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:08:12,245][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:08:12,758][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:08:13,254][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:08:13,757][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:08:14,253][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:08:14,753][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:08:15,256][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:08:15,756][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:08:16,257][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:08:16,776][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:08:17,273][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:08:17,773][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:08:18,272][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:08:18,771][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:08:19,286][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:08:19,783][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:08:20,286][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:08:20,783][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:08:21,278][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:08:21,785][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:08:22,284][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:08:22,784][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:08:23,291][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:08:23,789][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:08:24,287][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:08:24,787][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:08:25,288][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:08:25,792][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:08:26,293][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:08:26,797][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:08:27,297][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:08:27,795][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:08:28,295][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:08:28,794][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:08:29,293][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:08:29,796][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:08:30,293][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:08:30,794][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:08:31,291][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:08:31,789][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:08:32,314][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:08:32,815][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:08:33,318][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:08:33,818][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:08:34,316][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:08:34,818][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:08:35,317][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:08:35,815][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:08:36,317][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:08:36,816][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:08:37,315][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10252 tokens.
[2025-11-13 03:08:38,114][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 03:08:38,889][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:08:38,891][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:08:38,893][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:08:39,760][__main__][INFO] - Iteration 335 took 54s (33.21% Gen, 65.18% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 1m 30s. Estimated total time: 45h 0m 29s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 0s, 500 more iterations: 7h 30m 4s.
[2025-11-13 03:08:39,762][__main__][INFO] - Starting iteration 335.
[2025-11-13 03:08:40,246][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:08:40,247][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:08:58,008][__main__][INFO] - Number of regex retries in iteration 335: 0
[2025-11-13 03:08:58,008][__main__][INFO] - agents played in iteration 335 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:08:58,841][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:08:58,869][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:08:58,896][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:08:58,920][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:08:58,920][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:08:58,921][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:08:59,592][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:09:00,041][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:09:00,545][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:09:01,048][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:09:01,556][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:09:02,051][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:09:02,547][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:09:03,045][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:09:03,544][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:09:04,044][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:09:04,545][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:09:05,040][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:09:05,540][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:09:06,037][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:09:06,536][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:09:07,035][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:09:07,534][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:09:08,038][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:09:08,539][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:09:09,038][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:09:09,539][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:09:10,036][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:09:10,541][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:09:11,041][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:09:11,538][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:09:12,036][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:09:12,534][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:09:13,033][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:09:13,534][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:09:14,034][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:09:14,537][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:09:15,035][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:09:15,534][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:09:16,030][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:09:16,526][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:09:17,025][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:09:17,521][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:09:18,017][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:09:18,516][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:09:19,013][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:09:19,511][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:09:20,008][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:09:20,504][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:09:21,007][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:09:21,502][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:09:22,001][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:09:22,495][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:09:22,991][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:09:23,493][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:09:23,994][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:09:24,493][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:09:24,993][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:09:25,493][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:09:25,991][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:09:26,490][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:09:27,014][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:09:27,516][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:09:28,011][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:09:28,509][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:09:29,013][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:09:29,514][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:09:30,016][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:09:30,518][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:09:31,018][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:09:31,519][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10231 tokens.
[2025-11-13 03:09:32,298][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 03:09:33,035][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:09:33,036][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:09:33,038][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:09:33,909][__main__][INFO] - Iteration 336 took 53s (33.10% Gen, 65.28% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 43m 17s. Estimated total time: 44h 43m 9s. Time estimates for 10 more iterations: 8m 56s, 100 more iterations: 1h 29m 26s, 500 more iterations: 7h 27m 11s.
[2025-11-13 03:09:33,911][__main__][INFO] - Starting iteration 336.
[2025-11-13 03:09:34,381][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:09:34,382][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:09:50,180][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:09:51,797][__main__][INFO] - Number of regex retries in iteration 336: 1
[2025-11-13 03:09:51,798][__main__][INFO] - agents played in iteration 336 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:09:52,590][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:09:52,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:09:52,639][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:09:52,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:09:52,662][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:09:52,663][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:09:53,357][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:09:53,811][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:09:54,313][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:09:54,814][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:09:55,311][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:09:55,810][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:09:56,311][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:09:56,807][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:09:57,305][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:09:57,804][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:09:58,303][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:09:58,812][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:09:59,312][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:09:59,814][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:10:00,311][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:10:00,811][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:10:01,316][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:10:01,814][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:10:02,318][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:10:02,821][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:10:03,322][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:10:03,825][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:10:04,325][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:10:04,832][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:10:05,334][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:10:05,833][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:10:06,335][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:10:06,835][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:10:07,337][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:10:07,857][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:10:08,356][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:10:08,857][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:10:09,359][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:10:09,855][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:10:10,353][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:10:10,850][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:10:11,348][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:10:11,849][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:10:12,347][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:10:12,845][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:10:13,343][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:10:13,848][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:10:14,351][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:10:14,850][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:10:15,349][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:10:15,847][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:10:16,347][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:10:16,846][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:10:17,348][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:10:17,846][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:10:18,346][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:10:18,844][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:10:19,342][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:10:19,840][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:10:20,337][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:10:20,842][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:10:21,340][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:10:21,841][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:10:22,347][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:10:22,851][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:10:23,354][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:10:23,858][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:10:24,359][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:10:24,867][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:10:25,372][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10326 tokens.
[2025-11-13 03:10:26,170][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:32
[2025-11-13 03:10:26,905][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:10:26,907][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:10:26,908][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:10:27,843][__main__][INFO] - Iteration 337 took 53s (32.58% Gen, 65.67% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 32m 21s. Estimated total time: 44h 33m 7s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 6s, 500 more iterations: 7h 25m 31s.
[2025-11-13 03:10:27,845][__main__][INFO] - Starting iteration 337.
[2025-11-13 03:10:28,330][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:10:28,330][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:10:45,504][__main__][INFO] - Number of regex retries in iteration 337: 0
[2025-11-13 03:10:45,505][__main__][INFO] - agents played in iteration 337 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:10:46,383][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:10:46,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:10:46,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:10:46,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:10:46,464][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:10:46,464][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:10:47,318][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:10:47,785][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:10:48,303][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:10:48,802][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:10:49,298][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:10:49,819][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:10:50,315][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:10:50,811][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:10:51,321][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:10:51,817][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:10:52,312][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:10:52,807][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:10:53,302][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:10:53,812][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:10:54,309][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:10:54,805][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:10:55,309][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:10:55,814][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:10:56,315][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:10:56,814][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:10:57,313][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:10:57,825][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:10:58,327][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:10:58,826][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:10:59,330][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:10:59,829][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:11:00,333][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:11:00,834][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:11:01,336][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:11:01,834][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:11:02,357][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:11:02,860][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:11:03,360][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:11:03,858][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:11:04,356][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:11:04,853][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:11:05,352][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:11:05,853][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:11:06,351][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:11:06,852][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:11:07,349][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:11:07,852][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:11:08,352][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:11:08,854][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:11:09,352][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:11:09,854][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:11:10,354][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:11:10,853][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:11:11,351][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:11:11,854][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:11:12,354][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:11:12,857][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:11:13,359][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:11:13,857][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:11:14,356][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:11:14,858][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:11:15,354][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:11:15,853][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:11:16,359][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:11:16,859][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:11:17,362][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:11:17,864][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:11:18,360][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:11:18,861][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:11:19,359][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10180 tokens.
[2025-11-13 03:11:20,123][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-13 03:11:20,858][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:11:20,860][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:11:20,861][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:11:21,774][__main__][INFO] - Iteration 338 took 53s (32.13% Gen, 66.16% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 30m 35s. Estimated total time: 44h 32m 15s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 4s, 500 more iterations: 7h 25m 22s.
[2025-11-13 03:11:21,776][__main__][INFO] - Starting iteration 338.
[2025-11-13 03:11:22,262][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:11:22,262][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:11:41,060][__main__][INFO] - Number of regex retries in iteration 338: 0
[2025-11-13 03:11:41,061][__main__][INFO] - agents played in iteration 338 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:11:41,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:11:41,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:11:41,938][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:11:41,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:11:41,962][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:11:41,963][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:11:42,665][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:11:43,118][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:11:43,631][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:11:44,129][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:11:44,625][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:11:45,124][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:11:45,622][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:11:46,120][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:11:46,618][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:11:47,113][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:11:47,621][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:11:48,115][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:11:48,611][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:11:49,127][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:11:49,633][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:11:50,136][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:11:50,631][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:11:51,131][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:11:51,641][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:11:52,140][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:11:52,642][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:11:53,139][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:11:53,639][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:11:54,138][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:11:54,638][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:11:55,164][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:11:55,663][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:11:56,162][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:11:56,665][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:11:57,160][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:11:57,656][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:11:58,156][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:11:58,657][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:11:59,154][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:11:59,651][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:12:00,148][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:12:00,651][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:12:01,148][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:12:01,647][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:12:02,147][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:12:02,644][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:12:03,145][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:12:03,642][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:12:04,142][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:12:04,646][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:12:05,142][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:12:05,641][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:12:06,135][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:12:06,631][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:12:07,132][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:12:07,632][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:12:08,131][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:12:08,636][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:12:09,135][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:12:09,636][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:12:10,135][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:12:10,635][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:12:11,139][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:12:11,641][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:12:12,140][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:12:12,643][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:12:13,142][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:12:13,642][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:12:14,137][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:12:14,633][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10158 tokens.
[2025-11-13 03:12:15,368][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 61.95%, ΔTime: 00:00:32
[2025-11-13 03:12:16,132][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:12:16,133][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:12:16,135][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:12:17,123][__main__][INFO] - Iteration 339 took 54s (34.26% Gen, 63.93% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 40m 29s. Estimated total time: 45h 43m 5s. Time estimates for 10 more iterations: 9m 8s, 100 more iterations: 1h 31m 26s, 500 more iterations: 7h 37m 10s.
[2025-11-13 03:12:17,125][__main__][INFO] - Starting iteration 339.
[2025-11-13 03:12:17,599][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:12:17,600][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:12:33,349][__main__][INFO] - Number of regex retries in iteration 339: 0
[2025-11-13 03:12:33,350][__main__][INFO] - agents played in iteration 339 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:12:34,127][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:12:34,153][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:12:34,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:12:34,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:12:34,202][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:12:34,203][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:12:34,905][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:12:35,360][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:12:35,863][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:12:36,358][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:12:36,858][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:12:37,358][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:12:37,855][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:12:38,352][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:12:38,847][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:12:39,351][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:12:39,853][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:12:40,355][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:12:40,855][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:12:41,351][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:12:41,849][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:12:42,346][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:12:42,843][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:12:43,343][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:12:43,850][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:12:44,350][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:12:44,857][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:12:45,364][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:12:45,864][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:12:46,367][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:12:46,869][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:12:47,366][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:12:47,867][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:12:48,369][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:12:48,867][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:12:49,368][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:12:49,865][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:12:50,363][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:12:50,860][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:12:51,362][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:12:51,865][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:12:52,367][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:12:52,871][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:12:53,372][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:12:53,874][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:12:54,381][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:12:54,897][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:12:55,394][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:12:55,905][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:12:56,404][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:12:56,903][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:12:57,402][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:12:57,900][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:12:58,414][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:12:58,915][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:12:59,414][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:12:59,915][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:13:00,418][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:13:00,931][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:13:01,432][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:13:01,930][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:13:02,429][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:13:02,929][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:13:03,436][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:13:03,936][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:13:04,435][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:13:04,946][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:13:05,446][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:13:05,947][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:13:06,446][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:13:06,947][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10224 tokens.
[2025-11-13 03:13:07,672][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.40%, ΔTime: 00:00:32
[2025-11-13 03:13:08,426][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:13:08,427][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:13:08,429][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:13:09,596][__main__][INFO] - Iteration 340 took 51s (30.29% Gen, 67.46% Train). Generation: 15s, Training: 35s. Estimated remaining time: 38h 16m 23s. Estimated total time: 43h 19m 51s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 39s, 500 more iterations: 7h 13m 18s.
[2025-11-13 03:13:09,598][__main__][INFO] - Starting iteration 340.
[2025-11-13 03:13:10,081][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 03:13:10,081][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:13:14,077][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:13:17,137][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:13:26,364][__main__][INFO] - Number of regex retries in iteration 340: 2
[2025-11-13 03:13:26,365][__main__][INFO] - agents played in iteration 340 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:13:27,173][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:13:27,198][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:13:27,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:13:27,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:13:27,245][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:13:27,246][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:13:27,951][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:13:28,413][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:13:28,916][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:13:29,432][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:13:29,932][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:13:30,433][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:13:30,936][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:13:31,437][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:13:31,938][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:13:32,434][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:13:32,936][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:13:33,455][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:13:33,954][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:13:34,456][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:13:34,958][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:13:35,459][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:13:35,964][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:13:36,461][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:13:36,959][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:13:37,461][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:13:37,958][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:13:38,466][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:13:38,964][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:13:39,465][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:13:39,972][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:13:40,471][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:13:40,978][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:13:41,484][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:13:41,985][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:13:42,489][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:13:42,987][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:13:43,492][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:13:43,996][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:13:44,521][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:13:45,033][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:13:45,533][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:13:46,036][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:13:46,546][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:13:47,047][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:13:47,548][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:13:48,051][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:13:48,549][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:13:49,053][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:13:49,552][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:13:50,054][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:13:50,552][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:13:51,054][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:13:51,557][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:13:52,063][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:13:52,565][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:13:53,070][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:13:53,572][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:13:54,074][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:13:54,576][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:13:55,077][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:13:55,579][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:13:56,081][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:13:56,581][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:13:57,079][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:13:57,578][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:13:58,081][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:13:58,581][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:13:59,082][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:13:59,578][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:14:00,074][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10290 tokens.
[2025-11-13 03:14:00,800][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 03:14:01,565][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:14:01,567][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:14:01,568][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:14:03,425][__main__][INFO] - Iteration 341 took 53s (30.52% Gen, 65.99% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 22m 52s. Estimated total time: 44h 27m 14s. Time estimates for 10 more iterations: 8m 53s, 100 more iterations: 1h 28m 54s, 500 more iterations: 7h 24m 32s.
[2025-11-13 03:14:03,427][__main__][INFO] - Starting iteration 341.
[2025-11-13 03:14:03,896][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:14:03,897][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:14:22,076][__main__][INFO] - Number of regex retries in iteration 341: 0
[2025-11-13 03:14:22,077][__main__][INFO] - agents played in iteration 341 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:14:22,900][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:14:22,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:14:22,944][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:14:22,966][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:14:22,967][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:14:22,968][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:14:23,689][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:14:24,148][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:14:24,654][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:14:25,155][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:14:25,656][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:14:26,161][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:14:26,658][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:14:27,160][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:14:27,662][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:14:28,163][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:14:28,666][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:14:29,169][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:14:29,671][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:14:30,170][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:14:30,678][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:14:31,185][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:14:31,690][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:14:32,193][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:14:32,700][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:14:33,199][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:14:33,703][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:14:34,207][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:14:34,714][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:14:35,226][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:14:35,734][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:14:36,236][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:14:36,743][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:14:37,249][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:14:37,759][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:14:38,260][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:14:38,761][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:14:39,264][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:14:39,764][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:14:40,265][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:14:40,770][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:14:41,274][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:14:41,778][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:14:42,283][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:14:42,785][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:14:43,289][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:14:43,793][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:14:44,295][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:14:44,800][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:14:45,300][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:14:45,804][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:14:46,307][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:14:46,810][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:14:47,315][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:14:47,816][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:14:48,320][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:14:48,818][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:14:49,320][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:14:49,823][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:14:50,324][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:14:50,834][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:14:51,338][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:14:51,841][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:14:52,345][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:14:52,844][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:14:53,343][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:14:53,842][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:14:54,340][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:14:54,859][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:14:55,355][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:14:55,855][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10330 tokens.
[2025-11-13 03:14:56,598][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 03:14:57,368][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:14:57,370][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:14:57,372][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:14:58,313][__main__][INFO] - Iteration 342 took 54s (33.41% Gen, 64.86% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 15m 34s. Estimated total time: 45h 20m 51s. Time estimates for 10 more iterations: 9m 4s, 100 more iterations: 1h 30m 41s, 500 more iterations: 7h 33m 28s.
[2025-11-13 03:14:58,315][__main__][INFO] - Starting iteration 342.
[2025-11-13 03:14:58,816][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:14:58,817][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:15:15,789][__main__][INFO] - Number of regex retries in iteration 342: 0
[2025-11-13 03:15:15,790][__main__][INFO] - agents played in iteration 342 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:15:16,578][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:15:16,606][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:15:16,632][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:15:16,656][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:15:16,657][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:15:16,658][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:15:17,389][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:15:17,848][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:15:18,366][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:15:18,869][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:15:19,369][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:15:19,879][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:15:20,381][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:15:20,884][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:15:21,383][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:15:21,879][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:15:22,378][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:15:22,880][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:15:23,382][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:15:23,878][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:15:24,375][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:15:24,880][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:15:25,382][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:15:25,882][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:15:26,381][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:15:26,882][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:15:27,383][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:15:27,883][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:15:28,384][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:15:28,888][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:15:29,388][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:15:29,891][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:15:30,393][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:15:30,894][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:15:31,399][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:15:31,898][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:15:32,397][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:15:32,901][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:15:33,400][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:15:33,899][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:15:34,400][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:15:34,898][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:15:35,399][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:15:35,898][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:15:36,394][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:15:36,896][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:15:37,398][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:15:37,897][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:15:38,399][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:15:38,896][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:15:39,398][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:15:39,895][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:15:40,395][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:15:40,894][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:15:41,395][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:15:41,900][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:15:42,400][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:15:42,902][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:15:43,400][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:15:43,902][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:15:44,403][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:15:44,908][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:15:45,411][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:15:45,907][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:15:46,404][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:15:46,903][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:15:47,403][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:15:47,900][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:15:48,399][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:15:48,896][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:15:49,391][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10209 tokens.
[2025-11-13 03:15:50,159][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 03:15:50,927][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:15:50,929][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:15:50,930][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:15:51,891][__main__][INFO] - Iteration 343 took 53s (31.98% Gen, 66.21% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 7m 36s. Estimated total time: 44h 13m 46s. Time estimates for 10 more iterations: 8m 50s, 100 more iterations: 1h 28m 27s, 500 more iterations: 7h 22m 17s.
[2025-11-13 03:15:51,893][__main__][INFO] - Starting iteration 343.
[2025-11-13 03:15:52,362][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:15:52,362][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:15:55,644][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:16:06,143][__main__][INFO] - Number of regex retries in iteration 343: 1
[2025-11-13 03:16:06,144][__main__][INFO] - agents played in iteration 343 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:16:06,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:16:06,949][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:16:06,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:16:06,997][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:16:06,998][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:16:06,999][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:16:07,696][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:16:08,146][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:16:08,647][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:16:09,143][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:16:09,644][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:16:10,139][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:16:10,634][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:16:11,148][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:16:11,643][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:16:12,140][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:16:12,637][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:16:13,136][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:16:13,636][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:16:14,141][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:16:14,638][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:16:15,149][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:16:15,655][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:16:16,178][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:16:16,676][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:16:17,177][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:16:17,677][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:16:18,178][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:16:18,682][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:16:19,181][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:16:19,679][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:16:20,194][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:16:20,697][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:16:21,198][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:16:21,698][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:16:22,196][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:16:22,701][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:16:23,199][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:16:23,701][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:16:24,202][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:16:24,702][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:16:25,201][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:16:25,700][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:16:26,201][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:16:26,702][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:16:27,199][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:16:27,722][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:16:28,222][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:16:28,723][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:16:29,231][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:16:29,733][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:16:30,237][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:16:30,738][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:16:31,251][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:16:31,774][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:16:32,276][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:16:32,776][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:16:33,276][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:16:33,774][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:16:34,284][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:16:34,787][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:16:35,289][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:16:35,797][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:16:36,297][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:16:36,799][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:16:37,297][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:16:37,796][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:16:38,301][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:16:38,801][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:16:39,300][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:16:39,808][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10225 tokens.
[2025-11-13 03:16:40,583][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 03:16:41,372][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:16:41,375][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:16:41,376][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:16:42,345][__main__][INFO] - Iteration 344 took 49s (27.57% Gen, 70.49% Train). Generation: 13s, Training: 35s. Estimated remaining time: 36h 32m 9s. Estimated total time: 41h 39m 9s. Time estimates for 10 more iterations: 8m 19s, 100 more iterations: 1h 23m 18s, 500 more iterations: 6h 56m 31s.
[2025-11-13 03:16:42,347][__main__][INFO] - Starting iteration 344.
[2025-11-13 03:16:42,819][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:16:42,819][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:16:58,076][__main__][INFO] - Number of regex retries in iteration 344: 0
[2025-11-13 03:16:58,077][__main__][INFO] - agents played in iteration 344 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:16:58,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:16:58,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:16:58,957][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:16:58,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:16:58,980][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:16:58,981][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:16:59,722][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:17:00,178][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:17:00,681][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:17:01,183][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:17:01,689][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:17:02,197][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:17:02,698][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:17:03,201][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:17:03,706][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:17:04,211][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:17:04,715][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:17:05,217][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:17:05,722][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:17:06,225][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:17:06,726][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:17:07,233][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:17:07,739][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:17:08,238][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:17:08,740][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:17:09,239][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:17:09,740][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:17:10,241][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:17:10,740][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:17:11,239][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:17:11,742][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:17:12,243][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:17:12,748][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:17:13,251][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:17:13,752][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:17:14,266][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:17:14,768][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:17:15,274][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:17:15,781][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:17:16,281][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:17:16,785][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:17:17,285][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:17:17,788][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:17:18,293][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:17:18,793][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:17:19,307][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:17:19,807][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:17:20,312][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:17:20,820][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:17:21,323][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:17:21,835][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:17:22,339][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:17:22,841][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:17:23,347][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:17:23,848][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:17:24,350][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:17:24,851][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:17:25,356][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:17:25,858][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:17:26,358][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:17:26,860][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:17:27,374][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:17:27,876][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:17:28,374][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:17:28,873][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:17:29,374][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:17:29,881][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:17:30,384][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:17:30,882][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:17:31,385][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:17:31,882][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10241 tokens.
[2025-11-13 03:17:32,680][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.38%, ΔTime: 00:00:32
[2025-11-13 03:17:33,453][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:17:33,454][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:17:33,456][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:17:34,424][__main__][INFO] - Iteration 345 took 51s (29.57% Gen, 68.55% Train). Generation: 15s, Training: 35s. Estimated remaining time: 37h 52m 27s. Estimated total time: 43h 0m 19s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 0s, 500 more iterations: 7h 10m 3s.
[2025-11-13 03:17:34,427][__main__][INFO] - Starting iteration 345.
[2025-11-13 03:17:34,909][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:17:34,910][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:17:50,883][__main__][INFO] - Number of regex retries in iteration 345: 0
[2025-11-13 03:17:50,883][__main__][INFO] - agents played in iteration 345 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:17:51,742][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:17:51,768][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:17:51,793][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:17:51,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:17:51,817][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:17:51,818][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:17:52,553][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:17:53,014][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:17:53,520][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:17:54,022][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:17:54,526][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:17:55,031][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:17:55,532][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:17:56,035][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:17:56,536][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:17:57,036][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:17:57,537][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:17:58,034][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:17:58,534][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:17:59,033][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:17:59,535][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:18:00,039][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:18:00,536][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:18:01,036][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:18:01,537][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:18:02,034][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:18:02,532][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:18:03,029][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:18:03,529][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:18:04,027][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:18:04,530][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:18:05,031][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:18:05,529][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:18:06,029][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:18:06,532][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:18:07,032][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:18:07,534][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:18:08,038][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:18:08,541][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:18:09,049][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:18:09,556][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:18:10,058][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:18:10,561][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:18:11,065][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:18:11,566][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:18:12,069][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:18:12,573][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:18:13,078][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:18:13,581][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:18:14,082][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:18:14,586][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:18:15,088][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:18:15,591][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:18:16,092][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:18:16,594][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:18:17,106][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:18:17,603][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:18:18,100][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:18:18,616][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:18:19,115][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:18:19,613][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:18:20,114][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:18:20,613][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:18:21,146][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:18:21,652][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:18:22,153][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:18:22,659][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:18:23,156][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:18:23,658][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:18:24,160][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:18:24,664][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10269 tokens.
[2025-11-13 03:18:25,414][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.49%, ΔTime: 00:00:32
[2025-11-13 03:18:26,229][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:18:26,231][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:18:26,232][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:18:27,220][__main__][INFO] - Iteration 346 took 52s (30.54% Gen, 67.57% Train). Generation: 15s, Training: 35s. Estimated remaining time: 38h 26m 49s. Estimated total time: 43h 35m 34s. Time estimates for 10 more iterations: 8m 43s, 100 more iterations: 1h 27m 11s, 500 more iterations: 7h 15m 55s.
[2025-11-13 03:18:27,222][__main__][INFO] - Starting iteration 346.
[2025-11-13 03:18:27,714][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:18:27,715][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:18:43,209][__main__][INFO] - Number of regex retries in iteration 346: 0
[2025-11-13 03:18:43,209][__main__][INFO] - agents played in iteration 346 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:18:44,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:18:44,067][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:18:44,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:18:44,116][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:18:44,117][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:18:44,118][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:18:44,818][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:18:45,272][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:18:45,775][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:18:46,275][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:18:46,775][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:18:47,275][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:18:47,776][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:18:48,275][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:18:48,776][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:18:49,275][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:18:49,774][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:18:50,275][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:18:50,774][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:18:51,272][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:18:51,772][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:18:52,271][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:18:52,774][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:18:53,276][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:18:53,775][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:18:54,277][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:18:54,779][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:18:55,279][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:18:55,777][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:18:56,275][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:18:56,777][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:18:57,276][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:18:57,773][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:18:58,269][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:18:58,766][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:18:59,264][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:18:59,766][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:19:00,272][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:19:00,773][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:19:01,271][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:19:01,774][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:19:02,272][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:19:02,774][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:19:03,277][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:19:03,776][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:19:04,275][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:19:04,775][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:19:05,274][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:19:05,778][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:19:06,282][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:19:06,786][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:19:07,300][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:19:07,799][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:19:08,305][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:19:08,808][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:19:09,307][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:19:09,813][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:19:10,315][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:19:10,813][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:19:11,311][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:19:11,811][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:19:12,328][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:19:12,829][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:19:13,328][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:19:13,831][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:19:14,337][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:19:14,838][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:19:15,338][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:19:15,836][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:19:16,343][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:19:16,847][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10255 tokens.
[2025-11-13 03:19:17,625][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 03:19:18,423][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:19:18,425][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:19:18,427][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:19:19,369][__main__][INFO] - Iteration 347 took 51s (29.99% Gen, 68.18% Train). Generation: 15s, Training: 35s. Estimated remaining time: 37h 53m 7s. Estimated total time: 43h 2m 45s. Time estimates for 10 more iterations: 8m 36s, 100 more iterations: 1h 26m 5s, 500 more iterations: 7h 10m 27s.
[2025-11-13 03:19:19,371][__main__][INFO] - Starting iteration 347.
[2025-11-13 03:19:19,851][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:19:19,851][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:19:34,080][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given the values:
- I value hats at 1 and Bob values them at 10.
- I value books at 10 and Bob values them at 1.
- I value balls at 1 and Bob values them at 1.

Since Bob highly values hats and I do not, it's beneficial for me to claim all the hats to maximize my points. Books are less valuable to both of us compared to hats, so I propose to take none of those. Balls are valued the same by both, but since the allocation would be proportional and I do not gain a significant advantage by contesting them, I propose to take none as well. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:19:39,099][__main__][INFO] - Number of regex retries in iteration 347: 1
[2025-11-13 03:19:39,099][__main__][INFO] - agents played in iteration 347 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:19:39,936][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:19:39,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:19:39,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:19:40,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:19:40,012][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:19:40,013][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:19:40,787][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:19:41,251][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:19:41,763][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:19:42,266][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:19:42,765][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:19:43,266][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:19:43,768][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:19:44,277][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:19:44,778][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:19:45,279][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:19:45,780][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:19:46,282][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:19:46,782][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:19:47,285][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:19:47,798][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:19:48,299][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:19:48,803][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:19:49,297][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:19:49,795][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:19:50,294][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:19:50,791][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:19:51,289][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:19:51,791][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:19:52,293][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:19:52,796][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:19:53,295][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:19:53,796][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:19:54,309][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:19:54,810][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:19:55,319][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:19:55,819][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:19:56,321][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:19:56,839][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:19:57,341][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:19:57,845][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:19:58,348][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:19:58,852][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:19:59,359][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:19:59,862][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:20:00,364][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:20:00,870][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:20:01,371][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:20:01,874][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:20:02,374][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:20:02,875][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:20:03,375][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:20:03,874][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:20:04,374][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:20:04,881][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:20:05,379][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:20:05,882][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:20:06,380][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:20:06,883][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:20:07,387][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:20:07,889][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:20:08,391][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:20:08,891][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:20:09,394][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:20:09,898][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:20:10,398][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:20:10,897][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:20:11,403][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:20:11,901][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:20:12,402][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:20:12,900][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10263 tokens.
[2025-11-13 03:20:13,653][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 03:20:14,424][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:20:14,425][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:20:14,427][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:20:15,397][__main__][INFO] - Iteration 348 took 55s (34.65% Gen, 63.60% Train). Generation: 19s, Training: 35s. Estimated remaining time: 41h 6m 45s. Estimated total time: 46h 17m 18s. Time estimates for 10 more iterations: 9m 15s, 100 more iterations: 1h 32m 34s, 500 more iterations: 7h 42m 53s.
[2025-11-13 03:20:15,399][__main__][INFO] - Starting iteration 348.
[2025-11-13 03:20:15,885][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:20:15,885][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:20:20,197][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:20:31,820][__main__][INFO] - Number of regex retries in iteration 348: 1
[2025-11-13 03:20:31,820][__main__][INFO] - agents played in iteration 348 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:20:32,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:20:32,631][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:20:32,653][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:20:32,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:20:32,675][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:20:32,676][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:20:33,368][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:20:33,827][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:20:34,331][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:20:34,831][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:20:35,331][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:20:35,830][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:20:36,333][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:20:36,835][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:20:37,342][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:20:37,838][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:20:38,337][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:20:38,843][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:20:39,344][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:20:39,842][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:20:40,353][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:20:40,851][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:20:41,352][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:20:41,849][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:20:42,349][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:20:42,857][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:20:43,357][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:20:43,858][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:20:44,360][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:20:44,858][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:20:45,359][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:20:45,861][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:20:46,359][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:20:46,861][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:20:47,359][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:20:47,861][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:20:48,358][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:20:48,858][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:20:49,363][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:20:49,861][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:20:50,357][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:20:50,853][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:20:51,351][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:20:51,853][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:20:52,354][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:20:52,851][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:20:53,360][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:20:53,858][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:20:54,357][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:20:54,860][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:20:55,359][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:20:55,861][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:20:56,360][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:20:56,860][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:20:57,360][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:20:57,866][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:20:58,371][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:20:58,873][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:20:59,376][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:20:59,880][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:21:00,380][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:21:00,883][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:21:01,384][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:21:01,881][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:21:02,382][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:21:02,880][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:21:03,381][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:21:03,881][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:21:04,378][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:21:04,875][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:21:05,372][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10175 tokens.
[2025-11-13 03:21:06,124][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 03:21:06,903][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:21:06,904][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:21:06,908][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:21:07,897][__main__][INFO] - Iteration 349 took 52s (30.64% Gen, 67.46% Train). Generation: 15s, Training: 35s. Estimated remaining time: 38h 9m 12s. Estimated total time: 43h 20m 38s. Time estimates for 10 more iterations: 8m 40s, 100 more iterations: 1h 26m 41s, 500 more iterations: 7h 13m 26s.
[2025-11-13 03:21:07,899][__main__][INFO] - Starting iteration 349.
[2025-11-13 03:21:08,380][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:21:08,380][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:21:25,115][__main__][INFO] - Number of regex retries in iteration 349: 0
[2025-11-13 03:21:25,116][__main__][INFO] - agents played in iteration 349 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:21:25,900][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:21:25,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:21:25,944][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:21:25,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:21:25,981][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:21:25,982][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:21:26,661][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:21:27,112][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:21:27,629][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:21:28,124][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:21:28,628][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:21:29,124][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:21:29,622][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:21:30,144][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:21:30,645][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:21:31,143][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:21:31,642][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:21:32,140][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:21:32,655][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:21:33,155][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:21:33,661][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:21:34,159][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:21:34,660][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:21:35,164][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:21:35,667][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:21:36,169][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:21:36,672][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:21:37,174][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:21:37,676][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:21:38,177][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:21:38,678][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:21:39,177][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:21:39,677][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:21:40,176][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:21:40,677][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:21:41,179][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:21:41,678][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:21:42,177][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:21:42,675][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:21:43,176][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:21:43,676][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:21:44,181][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:21:44,679][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:21:45,176][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:21:45,676][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:21:46,175][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:21:46,677][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:21:47,179][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:21:47,678][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:21:48,183][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:21:48,680][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:21:49,181][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:21:49,685][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:21:50,189][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:21:50,688][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:21:51,188][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:21:51,687][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:21:52,191][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:21:52,696][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:21:53,195][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:21:53,694][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:21:54,192][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:21:54,693][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:21:55,190][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:21:55,689][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:21:56,192][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:21:56,690][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:21:57,188][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:21:57,687][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:21:58,183][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:21:58,682][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10175 tokens.
[2025-11-13 03:21:59,418][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.01%, Current % of VRAM taken: 60.27%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-13 03:22:00,191][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:22:00,193][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:22:00,196][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:22:01,160][__main__][INFO] - Iteration 350 took 52s (31.71% Gen, 66.46% Train). Generation: 16s, Training: 35s. Estimated remaining time: 38h 46m 43s. Estimated total time: 43h 59m 3s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 58s, 500 more iterations: 7h 19m 50s.
[2025-11-13 03:22:01,162][__main__][INFO] - Starting iteration 350.
[2025-11-13 03:22:01,650][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 03:22:01,650][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:22:19,376][__main__][INFO] - Number of regex retries in iteration 350: 0
[2025-11-13 03:22:19,376][__main__][INFO] - agents played in iteration 350 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:22:20,191][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:22:20,215][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:22:20,241][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:22:20,263][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:22:20,263][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:22:20,264][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:22:20,961][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:22:21,414][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:22:21,919][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:22:22,431][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:22:22,927][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:22:23,428][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:22:23,932][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:22:24,432][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:22:24,931][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:22:25,433][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:22:25,939][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:22:26,458][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:22:26,982][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:22:27,485][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:22:27,988][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:22:28,492][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:22:28,996][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:22:29,497][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:22:29,997][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:22:30,501][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:22:31,001][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:22:31,502][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:22:32,006][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:22:32,505][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:22:33,004][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:22:33,505][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:22:34,004][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:22:34,502][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:22:35,001][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:22:35,501][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:22:36,000][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:22:36,500][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:22:36,999][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:22:37,498][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:22:38,002][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:22:38,500][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:22:38,999][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:22:39,497][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:22:39,999][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:22:40,502][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:22:41,005][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:22:41,503][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:22:42,005][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:22:42,503][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:22:43,004][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:22:43,508][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:22:44,015][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:22:44,523][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:22:45,022][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:22:45,522][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:22:46,023][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:22:46,526][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:22:47,027][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:22:47,542][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:22:48,042][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:22:48,545][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:22:49,051][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:22:49,550][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:22:50,071][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:22:50,572][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:22:51,079][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:22:51,589][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:22:52,090][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:22:52,589][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:22:53,089][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10272 tokens.
[2025-11-13 03:22:53,840][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 03:22:54,620][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:22:54,622][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:22:54,624][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:22:56,878][__main__][INFO] - Iteration 351 took 55s (32.09% Gen, 63.82% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 48m 12s. Estimated total time: 46h 1m 27s. Time estimates for 10 more iterations: 9m 12s, 100 more iterations: 1h 32m 2s, 500 more iterations: 7h 40m 14s.
[2025-11-13 03:22:56,941][__main__][INFO] - Starting iteration 351.
[2025-11-13 03:22:57,431][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:22:57,432][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:23:15,142][__main__][INFO] - Number of regex retries in iteration 351: 0
[2025-11-13 03:23:15,143][__main__][INFO] - agents played in iteration 351 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:23:15,926][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:23:15,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:23:15,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:23:16,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:23:16,004][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:23:16,005][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:23:16,694][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:23:17,147][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:23:17,661][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:23:18,158][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:23:18,658][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:23:19,154][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:23:19,652][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:23:20,146][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:23:20,648][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:23:21,145][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:23:21,666][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:23:22,164][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:23:22,661][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:23:23,167][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:23:23,668][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:23:24,171][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:23:24,673][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:23:25,202][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:23:25,705][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:23:26,205][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:23:26,705][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:23:27,206][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:23:27,710][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:23:28,213][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:23:28,714][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:23:29,215][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:23:29,718][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:23:30,217][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:23:30,719][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:23:31,219][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:23:31,718][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:23:32,223][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:23:32,723][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:23:33,221][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:23:33,723][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:23:34,228][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:23:34,734][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:23:35,232][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:23:35,732][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:23:36,248][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:23:36,752][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:23:37,255][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:23:37,754][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:23:38,253][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:23:38,765][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:23:39,263][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:23:39,762][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:23:40,266][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:23:40,765][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:23:41,271][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:23:41,773][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:23:42,274][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:23:42,782][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:23:43,286][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:23:43,792][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:23:44,295][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:23:44,796][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:23:45,299][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:23:45,795][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:23:46,292][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:23:46,790][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:23:47,288][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:23:47,789][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:23:48,287][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:23:48,783][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10216 tokens.
[2025-11-13 03:23:49,517][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 03:23:50,304][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:23:50,306][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:23:50,307][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:23:51,257][__main__][INFO] - Iteration 352 took 53s (32.90% Gen, 65.33% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 37m 10s. Estimated total time: 44h 51m 20s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 42s, 500 more iterations: 7h 28m 33s.
[2025-11-13 03:23:51,259][__main__][INFO] - Starting iteration 352.
[2025-11-13 03:23:51,763][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:23:51,763][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:23:57,502][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:24:10,343][__main__][INFO] - Number of regex retries in iteration 352: 1
[2025-11-13 03:24:10,344][__main__][INFO] - agents played in iteration 352 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:24:11,176][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:24:11,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:24:11,227][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:24:11,250][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:24:11,250][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:24:11,251][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:24:11,953][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:24:12,410][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:24:12,917][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:24:13,417][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:24:13,918][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:24:14,417][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:24:14,916][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:24:15,418][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:24:15,919][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:24:16,416][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:24:16,914][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:24:17,416][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:24:17,918][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:24:18,424][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:24:18,924][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:24:19,428][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:24:19,930][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:24:20,434][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:24:20,937][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:24:21,438][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:24:21,953][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:24:22,457][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:24:22,960][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:24:23,476][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:24:23,977][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:24:24,476][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:24:24,975][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:24:25,472][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:24:25,973][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:24:26,472][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:24:26,972][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:24:27,472][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:24:27,968][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:24:28,464][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:24:28,961][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:24:29,460][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:24:29,975][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:24:30,474][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:24:30,976][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:24:31,474][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:24:31,975][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:24:32,482][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:24:32,985][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:24:33,489][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:24:33,996][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:24:34,498][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:24:34,999][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:24:35,499][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:24:35,999][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:24:36,501][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:24:37,005][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:24:37,505][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:24:38,006][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:24:38,506][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:24:39,012][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:24:39,514][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:24:40,015][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:24:40,517][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:24:41,016][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:24:41,514][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:24:42,010][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:24:42,508][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:24:43,005][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:24:43,502][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:24:43,998][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10173 tokens.
[2025-11-13 03:24:44,725][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 03:24:45,528][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:24:45,530][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:24:45,532][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:24:46,481][__main__][INFO] - Iteration 353 took 54s (33.96% Gen, 64.31% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 20m 51s. Estimated total time: 45h 35m 56s. Time estimates for 10 more iterations: 9m 7s, 100 more iterations: 1h 31m 11s, 500 more iterations: 7h 35m 59s.
[2025-11-13 03:24:46,483][__main__][INFO] - Starting iteration 353.
[2025-11-13 03:24:46,965][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:24:46,965][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:25:04,334][__main__][INFO] - Number of regex retries in iteration 353: 0
[2025-11-13 03:25:04,335][__main__][INFO] - agents played in iteration 353 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:25:05,124][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:25:05,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:25:05,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:25:05,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:25:05,200][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:25:05,201][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:25:05,883][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:25:06,335][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:25:06,836][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:25:07,336][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:25:07,835][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:25:08,336][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:25:08,832][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:25:09,327][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:25:09,823][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:25:10,320][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:25:10,819][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:25:11,318][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:25:11,818][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:25:12,317][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:25:12,815][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:25:13,316][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:25:13,816][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:25:14,321][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:25:14,827][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:25:15,329][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:25:15,854][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:25:16,359][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:25:16,865][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:25:17,366][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:25:17,868][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:25:18,368][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:25:18,867][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:25:19,364][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:25:19,863][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:25:20,362][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:25:20,877][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:25:21,378][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:25:21,878][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:25:22,378][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:25:22,876][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:25:23,373][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:25:23,871][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:25:24,368][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:25:24,880][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:25:25,377][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:25:25,875][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:25:26,375][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:25:26,874][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:25:27,386][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:25:27,884][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:25:28,387][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:25:28,900][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:25:29,403][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:25:29,905][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:25:30,403][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:25:30,903][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:25:31,412][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:25:31,912][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:25:32,414][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:25:32,919][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:25:33,420][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:25:33,925][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:25:34,428][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:25:34,929][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:25:35,430][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:25:35,928][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:25:36,426][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:25:36,925][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:25:37,422][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:25:37,920][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10146 tokens.
[2025-11-13 03:25:38,643][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 03:25:39,417][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:25:39,418][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:25:39,420][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:25:40,500][__main__][INFO] - Iteration 354 took 53s (32.44% Gen, 65.54% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 20m 48s. Estimated total time: 44h 36m 47s. Time estimates for 10 more iterations: 8m 55s, 100 more iterations: 1h 29m 13s, 500 more iterations: 7h 26m 7s.
[2025-11-13 03:25:40,502][__main__][INFO] - Starting iteration 354.
[2025-11-13 03:25:40,970][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:25:40,970][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:25:48,678][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:25:57,519][__main__][INFO] - Number of regex retries in iteration 354: 1
[2025-11-13 03:25:57,520][__main__][INFO] - agents played in iteration 354 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:25:58,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:25:58,321][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:25:58,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:25:58,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:25:58,369][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:25:58,369][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:25:59,061][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:25:59,511][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:26:00,018][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:26:00,514][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:26:01,011][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:26:01,510][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:26:02,005][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:26:02,504][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:26:03,007][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:26:03,512][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:26:04,015][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:26:04,514][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:26:05,013][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:26:05,515][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:26:06,040][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:26:06,539][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:26:07,038][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:26:07,539][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:26:08,040][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:26:08,541][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:26:09,042][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:26:09,543][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:26:10,045][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:26:10,542][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:26:11,045][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:26:11,545][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:26:12,044][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:26:12,544][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:26:13,044][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:26:13,545][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:26:14,044][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:26:14,544][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:26:15,047][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:26:15,543][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:26:16,045][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:26:16,540][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:26:17,044][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:26:17,543][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:26:18,039][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:26:18,538][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:26:19,039][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:26:19,541][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:26:20,040][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:26:20,542][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:26:21,045][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:26:21,550][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:26:22,054][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:26:22,557][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:26:23,054][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:26:23,558][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:26:24,060][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:26:24,562][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:26:25,060][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:26:25,574][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:26:26,077][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:26:26,591][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:26:27,097][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:26:27,595][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:26:28,096][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:26:28,593][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:26:29,101][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:26:29,614][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:26:30,124][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:26:30,634][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:26:31,139][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10170 tokens.
[2025-11-13 03:26:31,936][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 62.02%, ΔTime: 00:00:32
[2025-11-13 03:26:32,704][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:26:32,706][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:26:32,707][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:26:33,742][__main__][INFO] - Iteration 355 took 52s (31.36% Gen, 66.68% Train). Generation: 16s, Training: 35s. Estimated remaining time: 38h 41m 44s. Estimated total time: 43h 58m 36s. Time estimates for 10 more iterations: 8m 47s, 100 more iterations: 1h 27m 57s, 500 more iterations: 7h 19m 46s.
[2025-11-13 03:26:33,744][__main__][INFO] - Starting iteration 355.
[2025-11-13 03:26:34,237][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:26:34,237][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:26:52,085][__main__][INFO] - Number of regex retries in iteration 355: 0
[2025-11-13 03:26:52,086][__main__][INFO] - agents played in iteration 355 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:26:52,880][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:26:52,903][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:26:52,925][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:26:52,947][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:26:52,947][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:26:52,948][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:26:53,633][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:26:54,096][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:26:54,599][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:26:55,100][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:26:55,607][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:26:56,105][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:26:56,607][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:26:57,104][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:26:57,605][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:26:58,118][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:26:58,620][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:26:59,124][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:26:59,624][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:27:00,127][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:27:00,628][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:27:01,126][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:27:01,624][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:27:02,131][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:27:02,627][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:27:03,126][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:27:03,625][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:27:04,124][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:27:04,629][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:27:05,128][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:27:05,626][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:27:06,127][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:27:06,634][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:27:07,135][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:27:07,636][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:27:08,135][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:27:08,637][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:27:09,134][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:27:09,634][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:27:10,135][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:27:10,636][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:27:11,135][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:27:11,633][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:27:12,133][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:27:12,637][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:27:13,142][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:27:13,645][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:27:14,144][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:27:14,642][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:27:15,146][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:27:15,651][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:27:16,157][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:27:16,658][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:27:17,161][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:27:17,660][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:27:18,158][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:27:18,660][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:27:19,163][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:27:19,661][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:27:20,163][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:27:20,665][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:27:21,168][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:27:21,669][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:27:22,174][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:27:22,677][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:27:23,188][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:27:23,692][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:27:24,212][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:27:24,716][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:27:25,215][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:27:25,714][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10254 tokens.
[2025-11-13 03:27:26,445][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:32
[2025-11-13 03:27:27,235][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:27:27,236][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:27:27,238][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:27:28,216][__main__][INFO] - Iteration 356 took 53s (33.06% Gen, 65.12% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 41m 14s. Estimated total time: 44h 59m 1s. Time estimates for 10 more iterations: 8m 59s, 100 more iterations: 1h 29m 58s, 500 more iterations: 7h 29m 50s.
[2025-11-13 03:27:28,219][__main__][INFO] - Starting iteration 356.
[2025-11-13 03:27:28,723][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:27:28,724][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:27:48,058][__main__][INFO] - Number of regex retries in iteration 356: 0
[2025-11-13 03:27:48,058][__main__][INFO] - agents played in iteration 356 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:27:48,920][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:27:48,956][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:27:48,982][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:27:49,004][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:27:49,005][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:27:49,006][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:27:49,742][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:27:50,197][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:27:50,706][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:27:51,204][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:27:51,720][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:27:52,218][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:27:52,719][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:27:53,219][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:27:53,718][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:27:54,219][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:27:54,718][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:27:55,222][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:27:55,732][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:27:56,233][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:27:56,735][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:27:57,230][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:27:57,728][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:27:58,234][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:27:58,736][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:27:59,243][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:27:59,748][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:28:00,251][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:28:00,752][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:28:01,254][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:28:01,756][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:28:02,259][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:28:02,762][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:28:03,261][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:28:03,763][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:28:04,267][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:28:04,766][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:28:05,271][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:28:05,767][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:28:06,269][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:28:06,792][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:28:07,297][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:28:07,799][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:28:08,301][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:28:08,805][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:28:09,306][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:28:09,806][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:28:10,310][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:28:10,813][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:28:11,314][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:28:11,818][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:28:12,317][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:28:12,832][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:28:13,333][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:28:13,837][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:28:14,335][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:28:14,835][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:28:15,338][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:28:15,838][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:28:16,338][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:28:16,842][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:28:17,344][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:28:17,849][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:28:18,347][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:28:18,847][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:28:19,351][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:28:19,855][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:28:20,357][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:28:20,855][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:28:21,354][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:28:21,856][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10285 tokens.
[2025-11-13 03:28:22,582][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 03:28:23,362][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:28:23,364][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:28:23,366][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:28:24,422][__main__][INFO] - Iteration 357 took 55s (34.71% Gen, 63.39% Train). Generation: 19s, Training: 35s. Estimated remaining time: 41h 6m 16s. Estimated total time: 46h 24m 59s. Time estimates for 10 more iterations: 9m 16s, 100 more iterations: 1h 32m 49s, 500 more iterations: 7h 44m 9s.
[2025-11-13 03:28:24,425][__main__][INFO] - Starting iteration 357.
[2025-11-13 03:28:24,892][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:28:24,893][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:28:42,477][__main__][INFO] - Number of regex retries in iteration 357: 0
[2025-11-13 03:28:42,478][__main__][INFO] - agents played in iteration 357 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:28:43,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:28:43,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:28:43,361][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:28:43,384][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:28:43,384][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:28:43,385][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:28:44,139][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:28:44,594][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:28:45,098][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:28:45,604][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:28:46,104][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:28:46,602][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:28:47,136][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:28:47,634][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:28:48,135][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:28:48,638][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:28:49,143][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:28:49,644][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:28:50,144][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:28:50,644][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:28:51,141][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:28:51,640][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:28:52,139][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:28:52,634][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:28:53,132][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:28:53,650][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:28:54,150][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:28:54,650][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:28:55,149][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:28:55,648][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:28:56,150][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:28:56,652][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:28:57,153][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:28:57,655][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:28:58,153][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:28:58,654][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:28:59,156][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:28:59,654][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:29:00,156][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:29:00,657][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:29:01,157][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:29:01,654][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:29:02,153][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:29:02,659][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:29:03,159][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:29:03,656][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:29:04,158][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:29:04,659][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:29:05,159][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:29:05,660][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:29:06,159][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:29:06,657][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:29:07,159][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:29:07,658][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:29:08,160][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:29:08,658][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:29:09,156][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:29:09,656][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:29:10,157][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:29:10,659][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:29:11,158][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:29:11,660][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:29:12,164][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:29:12,659][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:29:13,159][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:29:13,654][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:29:14,153][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:29:14,660][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:29:15,159][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:29:15,658][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:29:16,154][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10190 tokens.
[2025-11-13 03:29:16,884][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:32
[2025-11-13 03:29:17,671][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:29:17,760][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:29:17,763][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:29:18,834][__main__][INFO] - Iteration 358 took 53s (32.60% Gen, 65.41% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 37m 29s. Estimated total time: 44h 57m 6s. Time estimates for 10 more iterations: 8m 59s, 100 more iterations: 1h 29m 54s, 500 more iterations: 7h 29m 31s.
[2025-11-13 03:29:18,836][__main__][INFO] - Starting iteration 358.
[2025-11-13 03:29:19,304][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:29:19,304][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:29:34,798][__main__][INFO] - Number of regex retries in iteration 358: 0
[2025-11-13 03:29:34,798][__main__][INFO] - agents played in iteration 358 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:29:35,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:29:35,721][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:29:35,749][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:29:35,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:29:35,772][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:29:35,774][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:29:36,546][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:29:37,004][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:29:37,511][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:29:38,013][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:29:38,517][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:29:39,020][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:29:39,522][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:29:40,037][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:29:40,539][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:29:41,057][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:29:41,561][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:29:42,058][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:29:42,557][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:29:43,056][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:29:43,560][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:29:44,057][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:29:44,553][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:29:45,052][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:29:45,552][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:29:46,056][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:29:46,553][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:29:47,054][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:29:47,562][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:29:48,063][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:29:48,564][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:29:49,070][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:29:49,574][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:29:50,076][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:29:50,580][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:29:51,083][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:29:51,593][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:29:52,095][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:29:52,598][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:29:53,097][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:29:53,596][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:29:54,100][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:29:54,601][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:29:55,099][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:29:55,598][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:29:56,099][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:29:56,594][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:29:57,092][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:29:57,592][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:29:58,094][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:29:58,594][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:29:59,098][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:29:59,595][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:30:00,092][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:30:00,593][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:30:01,091][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:30:01,589][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:30:02,092][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:30:02,595][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:30:03,096][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:30:03,597][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:30:04,097][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:30:04,599][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:30:05,095][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:30:05,594][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:30:06,093][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:30:06,593][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:30:07,096][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:30:07,594][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:30:08,091][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:30:08,591][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10222 tokens.
[2025-11-13 03:30:09,413][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.39%, ΔTime: 00:00:32
[2025-11-13 03:30:10,183][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:30:10,185][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:30:10,186][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:30:11,131][__main__][INFO] - Iteration 359 took 51s (29.89% Gen, 68.28% Train). Generation: 15s, Training: 35s. Estimated remaining time: 37h 50m 54s. Estimated total time: 43h 11m 24s. Time estimates for 10 more iterations: 8m 38s, 100 more iterations: 1h 26m 22s, 500 more iterations: 7h 11m 54s.
[2025-11-13 03:30:11,133][__main__][INFO] - Starting iteration 359.
[2025-11-13 03:30:11,625][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:30:11,626][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:30:28,300][__main__][INFO] - Number of regex retries in iteration 359: 0
[2025-11-13 03:30:28,300][__main__][INFO] - agents played in iteration 359 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:30:29,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:30:29,125][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:30:29,153][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:30:29,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:30:29,178][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:30:29,179][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:30:29,912][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:30:30,369][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:30:30,873][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:30:31,380][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:30:31,878][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:30:32,381][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:30:32,882][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:30:33,381][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:30:33,881][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:30:34,380][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:30:34,884][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:30:35,384][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:30:35,883][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:30:36,407][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:30:36,913][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:30:37,411][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:30:37,914][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:30:38,420][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:30:38,922][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:30:39,432][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:30:39,938][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:30:40,440][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:30:40,950][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:30:41,453][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:30:41,962][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:30:42,465][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:30:42,974][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:30:43,482][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:30:43,987][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:30:44,489][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:30:44,993][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:30:45,500][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:30:46,004][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:30:46,507][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:30:47,005][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:30:47,507][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:30:48,006][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:30:48,507][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:30:49,008][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:30:49,507][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:30:50,008][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:30:50,508][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:30:51,009][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:30:51,511][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:30:52,013][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:30:52,518][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:30:53,020][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:30:53,519][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:30:54,024][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:30:54,530][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:30:55,032][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:30:55,550][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:30:56,050][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:30:56,554][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:30:57,058][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:30:57,560][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:30:58,061][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:30:58,558][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:30:59,060][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:30:59,560][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:31:00,058][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:31:00,570][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:31:01,067][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:31:01,565][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:31:02,068][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10277 tokens.
[2025-11-13 03:31:02,842][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:32
[2025-11-13 03:31:03,872][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:31:03,873][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:31:03,894][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:31:05,096][__main__][INFO] - Iteration 360 took 53s (31.18% Gen, 66.57% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 12m 11s. Estimated total time: 44h 33m 34s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 7s, 500 more iterations: 7h 25m 35s.
[2025-11-13 03:31:05,098][__main__][INFO] - Starting iteration 360.
[2025-11-13 03:31:05,596][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 03:31:05,597][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:31:23,291][__main__][INFO] - Number of regex retries in iteration 360: 0
[2025-11-13 03:31:23,292][__main__][INFO] - agents played in iteration 360 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:31:24,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:31:24,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:31:24,190][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:31:24,217][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:31:24,218][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:31:24,218][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:31:24,984][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:31:25,442][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:31:25,946][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:31:26,448][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:31:26,959][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:31:27,457][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:31:27,954][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:31:28,454][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:31:28,954][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:31:29,469][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:31:29,973][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:31:30,472][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:31:30,981][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:31:31,502][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:31:32,008][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:31:32,507][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:31:33,005][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:31:33,502][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:31:33,997][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:31:34,495][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:31:34,992][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:31:35,487][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:31:35,984][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:31:36,480][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:31:36,979][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:31:37,482][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:31:37,980][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:31:38,482][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:31:38,984][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:31:39,483][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:31:39,982][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:31:40,483][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:31:40,984][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:31:41,493][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:31:41,994][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:31:42,496][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:31:42,996][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:31:43,500][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:31:43,999][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:31:44,501][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:31:45,004][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:31:45,513][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:31:46,012][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:31:46,516][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:31:47,018][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:31:47,518][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:31:48,018][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:31:48,518][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:31:49,018][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:31:49,518][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:31:50,017][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:31:50,516][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:31:51,015][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:31:51,514][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:31:52,013][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:31:52,511][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:31:53,009][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:31:53,509][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:31:54,006][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:31:54,504][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:31:55,000][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:31:55,500][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:31:56,003][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:31:56,502][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:31:56,998][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10095 tokens.
[2025-11-13 03:31:57,754][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 03:31:58,564][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:31:58,566][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:31:58,567][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:32:00,601][__main__][INFO] - Iteration 361 took 55s (32.17% Gen, 64.13% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 27m 58s. Estimated total time: 45h 50m 17s. Time estimates for 10 more iterations: 9m 10s, 100 more iterations: 1h 31m 40s, 500 more iterations: 7h 38m 22s.
[2025-11-13 03:32:00,603][__main__][INFO] - Starting iteration 361.
[2025-11-13 03:32:01,085][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:32:01,085][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:32:18,460][__main__][INFO] - Number of regex retries in iteration 361: 0
[2025-11-13 03:32:18,460][__main__][INFO] - agents played in iteration 361 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:32:19,293][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:32:19,329][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:32:19,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:32:19,377][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:32:19,378][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:32:19,379][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:32:20,157][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:32:20,613][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:32:21,121][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:32:21,623][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:32:22,122][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:32:22,623][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:32:23,125][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:32:23,627][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:32:24,130][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:32:24,627][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:32:25,123][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:32:25,621][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:32:26,119][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:32:26,620][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:32:27,119][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:32:27,616][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:32:28,117][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:32:28,620][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:32:29,120][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:32:29,617][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:32:30,117][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:32:30,616][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:32:31,114][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:32:31,617][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:32:32,116][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:32:32,619][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:32:33,122][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:32:33,624][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:32:34,135][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:32:34,637][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:32:35,139][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:32:35,643][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:32:36,148][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:32:36,652][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:32:37,153][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:32:37,654][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:32:38,164][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:32:38,667][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:32:39,173][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:32:39,673][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:32:40,172][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:32:40,671][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:32:41,170][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:32:41,668][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:32:42,167][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:32:42,667][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:32:43,165][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:32:43,663][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:32:44,164][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:32:44,673][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:32:45,169][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:32:45,672][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:32:46,169][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:32:46,665][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:32:47,168][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:32:47,664][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:32:48,160][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:32:48,660][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:32:49,156][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:32:49,657][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:32:50,156][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:32:50,653][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:32:51,153][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:32:51,650][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:32:52,150][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10190 tokens.
[2025-11-13 03:32:52,903][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 03:32:53,681][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:32:53,683][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:32:53,685][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:32:54,789][__main__][INFO] - Iteration 362 took 53s (32.35% Gen, 65.59% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 22m 2s. Estimated total time: 44h 45m 15s. Time estimates for 10 more iterations: 8m 57s, 100 more iterations: 1h 29m 30s, 500 more iterations: 7h 27m 32s.
[2025-11-13 03:32:54,792][__main__][INFO] - Starting iteration 362.
[2025-11-13 03:32:55,301][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:32:55,301][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:33:05,305][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:33:14,146][__main__][INFO] - Number of regex retries in iteration 362: 1
[2025-11-13 03:33:14,147][__main__][INFO] - agents played in iteration 362 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:33:14,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:33:14,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:33:14,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:33:15,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:33:15,010][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:33:15,010][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:33:15,755][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:33:16,208][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:33:16,719][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:33:17,222][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:33:17,723][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:33:18,220][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:33:18,719][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:33:19,217][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:33:19,721][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:33:20,218][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:33:20,716][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:33:21,218][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:33:21,718][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:33:22,222][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:33:22,721][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:33:23,218][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:33:23,720][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:33:24,220][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:33:24,725][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:33:25,223][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:33:25,720][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:33:26,221][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:33:26,724][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:33:27,223][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:33:27,726][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:33:28,224][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:33:28,726][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:33:29,226][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:33:29,733][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:33:30,238][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:33:30,743][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:33:31,245][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:33:31,754][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:33:32,255][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:33:32,759][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:33:33,270][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:33:33,772][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:33:34,280][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:33:34,781][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:33:35,281][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:33:35,782][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:33:36,283][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:33:36,784][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:33:37,292][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:33:37,794][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:33:38,311][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:33:38,812][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:33:39,311][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:33:39,814][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:33:40,310][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:33:40,807][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:33:41,303][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:33:41,800][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:33:42,312][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:33:42,812][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:33:43,317][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:33:43,815][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:33:44,313][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:33:44,817][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:33:45,315][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:33:45,814][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:33:46,313][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:33:46,811][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:33:47,310][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:33:47,810][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10171 tokens.
[2025-11-13 03:33:48,574][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:32
[2025-11-13 03:33:49,351][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:33:49,352][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:33:49,354][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:33:50,454][__main__][INFO] - Iteration 363 took 55s (34.17% Gen, 63.83% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 33m 32s. Estimated total time: 45h 57m 41s. Time estimates for 10 more iterations: 9m 11s, 100 more iterations: 1h 31m 55s, 500 more iterations: 7h 39m 36s.
[2025-11-13 03:33:50,456][__main__][INFO] - Starting iteration 363.
[2025-11-13 03:33:50,944][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:33:50,945][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:34:08,687][__main__][INFO] - Number of regex retries in iteration 363: 0
[2025-11-13 03:34:08,687][__main__][INFO] - agents played in iteration 363 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:34:09,468][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:34:09,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:34:09,521][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:34:09,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:34:09,545][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:34:09,547][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:34:10,268][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:34:10,721][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:34:11,239][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:34:11,737][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:34:12,234][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:34:12,732][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:34:13,235][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:34:13,743][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:34:14,240][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:34:14,737][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:34:15,247][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:34:15,747][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:34:16,246][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:34:16,748][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:34:17,244][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:34:17,750][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:34:18,254][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:34:18,754][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:34:19,255][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:34:19,755][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:34:20,254][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:34:20,752][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:34:21,251][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:34:21,748][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:34:22,243][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:34:22,738][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:34:23,242][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:34:23,736][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:34:24,242][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:34:24,739][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:34:25,237][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:34:25,744][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:34:26,244][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:34:26,748][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:34:27,248][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:34:27,749][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:34:28,251][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:34:28,753][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:34:29,253][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:34:29,757][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:34:30,256][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:34:30,757][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:34:31,255][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:34:31,755][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:34:32,258][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:34:32,758][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:34:33,259][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:34:33,758][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:34:34,256][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:34:34,758][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:34:35,257][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:34:35,755][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:34:36,256][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:34:36,756][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:34:37,253][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:34:37,752][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:34:38,251][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:34:38,750][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:34:39,247][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:34:39,753][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:34:40,252][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:34:40,751][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:34:41,252][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:34:41,748][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:34:42,246][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10185 tokens.
[2025-11-13 03:34:43,017][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:32
[2025-11-13 03:34:43,790][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:34:43,792][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:34:43,795][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:34:44,830][__main__][INFO] - Iteration 364 took 53s (32.92% Gen, 65.15% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 29m 18s. Estimated total time: 44h 54m 21s. Time estimates for 10 more iterations: 8m 58s, 100 more iterations: 1h 29m 48s, 500 more iterations: 7h 29m 3s.
[2025-11-13 03:34:44,833][__main__][INFO] - Starting iteration 364.
[2025-11-13 03:34:45,344][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:34:45,344][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:34:53,008][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:35:03,109][__main__][INFO] - Number of regex retries in iteration 364: 1
[2025-11-13 03:35:03,110][__main__][INFO] - agents played in iteration 364 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:35:03,970][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:35:03,993][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:35:04,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:35:04,038][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:35:04,038][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:35:04,039][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:35:04,768][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:35:05,223][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:35:05,726][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:35:06,236][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:35:06,735][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:35:07,239][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:35:07,739][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:35:08,241][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:35:08,739][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:35:09,238][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:35:09,738][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:35:10,243][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:35:10,744][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:35:11,245][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:35:11,754][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:35:12,260][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:35:12,762][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:35:13,257][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:35:13,755][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:35:14,255][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:35:14,751][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:35:15,249][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:35:15,743][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:35:16,237][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:35:16,741][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:35:17,238][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:35:17,734][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:35:18,235][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:35:18,734][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:35:19,235][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:35:19,736][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:35:20,237][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:35:20,755][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:35:21,259][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:35:21,760][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:35:22,265][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:35:22,766][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:35:23,277][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:35:23,778][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:35:24,280][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:35:24,783][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:35:25,286][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:35:25,785][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:35:26,287][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:35:26,792][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:35:27,291][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:35:27,793][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:35:28,296][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:35:28,800][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:35:29,299][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:35:29,797][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:35:30,295][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:35:30,795][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:35:31,296][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:35:31,794][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:35:32,292][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:35:32,791][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:35:33,298][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:35:33,801][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:35:34,301][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:35:34,801][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:35:35,304][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:35:35,807][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:35:36,310][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:35:36,810][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10215 tokens.
[2025-11-13 03:35:37,581][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 03:35:38,590][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:35:38,591][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:35:38,610][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:35:39,979][__main__][INFO] - Iteration 365 took 54s (32.52% Gen, 64.98% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 5m 50s. Estimated total time: 45h 31m 49s. Time estimates for 10 more iterations: 9m 6s, 100 more iterations: 1h 31m 3s, 500 more iterations: 7h 35m 18s.
[2025-11-13 03:35:39,982][__main__][INFO] - Starting iteration 365.
[2025-11-13 03:35:40,472][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:35:40,473][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:35:57,816][__main__][INFO] - Number of regex retries in iteration 365: 0
[2025-11-13 03:35:57,817][__main__][INFO] - agents played in iteration 365 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:35:58,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:35:58,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:35:58,699][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:35:58,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:35:58,723][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:35:58,724][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:35:59,445][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:35:59,896][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:36:00,406][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:36:00,905][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:36:01,410][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:36:01,914][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:36:02,419][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:36:02,922][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:36:03,421][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:36:03,928][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:36:04,431][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:36:04,932][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:36:05,434][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:36:05,935][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:36:06,432][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:36:06,930][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:36:07,427][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:36:07,939][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:36:08,436][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:36:08,932][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:36:09,448][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:36:09,948][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:36:10,447][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:36:10,946][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:36:11,442][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:36:11,955][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:36:12,454][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:36:12,957][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:36:13,459][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:36:13,962][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:36:14,467][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:36:14,973][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:36:15,478][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:36:15,982][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:36:16,483][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:36:16,985][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:36:17,486][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:36:17,988][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:36:18,498][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:36:18,998][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:36:19,501][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:36:20,004][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:36:20,504][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:36:21,016][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:36:21,517][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:36:22,019][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:36:22,516][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:36:23,016][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:36:23,523][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:36:24,022][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:36:24,520][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:36:25,022][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:36:25,522][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:36:26,020][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:36:26,519][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:36:27,018][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:36:27,543][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:36:28,042][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:36:28,540][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:36:29,038][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:36:29,541][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:36:30,045][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:36:30,542][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:36:31,040][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:36:31,540][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10144 tokens.
[2025-11-13 03:36:32,351][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:32
[2025-11-13 03:36:33,133][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:36:33,135][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:36:33,136][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:36:34,253][__main__][INFO] - Iteration 366 took 53s (32.25% Gen, 65.67% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 22m 14s. Estimated total time: 44h 49m 7s. Time estimates for 10 more iterations: 8m 57s, 100 more iterations: 1h 29m 38s, 500 more iterations: 7h 28m 11s.
[2025-11-13 03:36:34,256][__main__][INFO] - Starting iteration 366.
[2025-11-13 03:36:34,740][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:36:34,741][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:36:53,840][__main__][INFO] - Number of regex retries in iteration 366: 0
[2025-11-13 03:36:53,840][__main__][INFO] - agents played in iteration 366 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:36:54,673][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:36:54,698][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:36:54,724][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:36:54,746][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:36:54,746][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:36:54,747][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:36:55,473][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:36:56,083][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:36:56,593][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:36:57,097][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:36:57,602][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:36:58,107][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:36:58,610][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:36:59,118][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:36:59,621][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:37:00,122][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:37:00,629][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:37:01,136][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:37:01,639][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:37:02,137][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:37:02,635][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:37:03,151][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:37:03,650][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:37:04,148][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:37:04,644][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:37:05,143][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:37:05,640][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:37:06,136][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:37:06,634][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:37:07,133][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:37:07,636][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:37:08,137][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:37:08,637][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:37:09,140][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:37:09,658][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:37:10,161][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:37:10,662][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:37:11,158][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:37:11,661][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:37:12,164][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:37:12,667][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:37:13,172][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:37:13,676][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:37:14,179][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:37:14,680][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:37:15,181][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:37:15,681][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:37:16,181][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:37:16,676][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:37:17,176][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:37:17,674][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:37:18,169][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:37:18,671][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:37:19,165][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:37:19,664][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:37:20,160][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:37:20,659][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:37:21,156][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:37:21,653][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:37:22,149][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:37:22,653][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:37:23,153][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:37:23,650][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:37:24,149][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:37:24,647][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:37:25,144][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:37:25,644][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:37:26,140][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:37:26,655][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:37:27,154][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:37:27,651][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10276 tokens.
[2025-11-13 03:37:28,448][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.46%, ΔTime: 00:00:32
[2025-11-13 03:37:29,247][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:37:29,249][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:37:29,251][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:37:30,165][__main__][INFO] - Iteration 367 took 55s (34.46% Gen, 63.89% Train). Generation: 19s, Training: 35s. Estimated remaining time: 40h 43m 27s. Estimated total time: 46h 11m 15s. Time estimates for 10 more iterations: 9m 14s, 100 more iterations: 1h 32m 22s, 500 more iterations: 7h 41m 52s.
[2025-11-13 03:37:30,167][__main__][INFO] - Starting iteration 367.
[2025-11-13 03:37:30,655][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:37:30,656][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:37:51,190][__main__][INFO] - Number of regex retries in iteration 367: 0
[2025-11-13 03:37:51,191][__main__][INFO] - agents played in iteration 367 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:37:52,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:37:52,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:37:52,163][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:37:52,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:37:52,186][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:37:52,187][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:37:52,938][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:37:53,404][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:37:53,909][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:37:54,413][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:37:54,911][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:37:55,412][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:37:55,914][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:37:56,412][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:37:56,914][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:37:57,412][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:37:57,908][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:37:58,406][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:37:58,902][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:37:59,397][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:37:59,894][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:38:00,389][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:38:00,882][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:38:01,384][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:38:01,885][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:38:02,384][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:38:02,888][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:38:03,395][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:38:03,897][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:38:04,398][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:38:04,900][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:38:05,399][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:38:05,898][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:38:06,399][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:38:06,899][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:38:07,399][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:38:07,899][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:38:08,400][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:38:08,903][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:38:09,405][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:38:09,903][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:38:10,406][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:38:10,904][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:38:11,403][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:38:11,899][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:38:12,396][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:38:12,893][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:38:13,389][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:38:13,885][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:38:14,384][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:38:14,884][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:38:15,382][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:38:15,878][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:38:16,376][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:38:16,876][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:38:17,374][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:38:17,877][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:38:18,375][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:38:18,872][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:38:19,374][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:38:19,874][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:38:20,373][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:38:20,872][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:38:21,374][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:38:21,873][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:38:22,375][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:38:22,871][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:38:23,377][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:38:23,880][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:38:24,380][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:38:24,883][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10134 tokens.
[2025-11-13 03:38:25,639][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 03:38:26,446][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:38:26,447][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:38:26,449][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:38:27,374][__main__][INFO] - Iteration 368 took 56s (36.21% Gen, 62.16% Train). Generation: 20s, Training: 35s. Estimated remaining time: 41h 47m 12s. Estimated total time: 47h 15m 58s. Time estimates for 10 more iterations: 9m 27s, 100 more iterations: 1h 34m 31s, 500 more iterations: 7h 52m 39s.
[2025-11-13 03:38:27,376][__main__][INFO] - Starting iteration 368.
[2025-11-13 03:38:27,885][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:38:27,886][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:38:46,126][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:38:47,120][__main__][INFO] - Number of regex retries in iteration 368: 1
[2025-11-13 03:38:47,121][__main__][INFO] - agents played in iteration 368 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:38:47,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:38:47,940][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:38:47,976][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:38:48,005][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:38:48,006][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:38:48,007][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:38:48,718][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:38:49,177][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:38:49,681][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:38:50,177][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:38:50,679][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:38:51,175][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:38:51,675][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:38:52,171][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:38:52,671][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:38:53,169][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:38:53,667][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:38:54,171][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:38:54,672][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:38:55,171][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:38:55,670][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:38:56,171][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:38:56,671][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:38:57,169][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:38:57,672][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:38:58,174][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:38:58,678][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:38:59,174][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:38:59,672][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:39:00,172][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:39:00,670][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:39:01,175][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:39:01,675][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:39:02,176][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:39:02,686][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:39:03,185][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:39:03,685][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:39:04,185][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:39:04,683][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:39:05,190][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:39:05,688][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:39:06,186][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:39:06,686][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:39:07,183][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:39:07,682][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:39:08,183][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:39:08,683][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:39:09,185][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:39:09,682][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:39:10,177][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:39:10,677][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:39:11,176][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:39:11,677][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:39:12,178][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:39:12,675][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:39:13,177][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:39:13,679][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:39:14,182][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:39:14,683][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:39:15,183][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:39:15,682][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:39:16,180][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:39:16,681][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:39:17,177][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:39:17,675][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:39:18,179][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:39:18,679][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:39:19,177][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:39:19,677][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:39:20,175][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:39:20,676][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10123 tokens.
[2025-11-13 03:39:21,474][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 03:39:22,290][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:39:22,292][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:39:22,294][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:39:23,241][__main__][INFO] - Iteration 369 took 55s (34.75% Gen, 63.54% Train). Generation: 19s, Training: 35s. Estimated remaining time: 40h 38m 8s. Estimated total time: 46h 7m 50s. Time estimates for 10 more iterations: 9m 13s, 100 more iterations: 1h 32m 15s, 500 more iterations: 7h 41m 18s.
[2025-11-13 03:39:23,243][__main__][INFO] - Starting iteration 369.
[2025-11-13 03:39:23,733][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:39:23,733][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:39:42,449][__main__][INFO] - Number of regex retries in iteration 369: 0
[2025-11-13 03:39:42,450][__main__][INFO] - agents played in iteration 369 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:39:43,232][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:39:43,259][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:39:43,285][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:39:43,308][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:39:43,309][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:39:43,309][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:39:44,001][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:39:44,456][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:39:44,957][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:39:45,454][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:39:45,948][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:39:46,444][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:39:46,938][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:39:47,432][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:39:47,931][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:39:48,432][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:39:48,929][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:39:49,426][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:39:49,924][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:39:50,421][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:39:50,917][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:39:51,425][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:39:51,929][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:39:52,437][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:39:52,963][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:39:53,461][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:39:53,963][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:39:54,463][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:39:54,964][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:39:55,462][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:39:55,957][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:39:56,469][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:39:56,977][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:39:57,478][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:39:57,982][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:39:58,484][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:39:58,996][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:39:59,500][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:40:00,000][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:40:00,500][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:40:01,000][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:40:01,502][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:40:02,003][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:40:02,501][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:40:03,006][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:40:03,508][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:40:04,008][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:40:04,507][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:40:05,004][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:40:05,507][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:40:06,003][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:40:06,504][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:40:07,002][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:40:07,501][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:40:07,998][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:40:08,499][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:40:08,998][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:40:09,497][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:40:09,994][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:40:10,492][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:40:10,992][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:40:11,492][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:40:11,993][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:40:12,494][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:40:12,999][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:40:13,502][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:40:14,002][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:40:14,505][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:40:15,002][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:40:15,504][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:40:16,003][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10178 tokens.
[2025-11-13 03:40:16,792][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 03:40:17,577][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:40:17,579][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:40:17,581][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:40:18,514][__main__][INFO] - Iteration 370 took 54s (34.16% Gen, 64.13% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 8m 30s. Estimated total time: 45h 39m 6s. Time estimates for 10 more iterations: 9m 7s, 100 more iterations: 1h 31m 18s, 500 more iterations: 7h 36m 31s.
[2025-11-13 03:40:18,517][__main__][INFO] - Starting iteration 370.
[2025-11-13 03:40:19,003][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 03:40:19,004][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:40:24,012][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:40:36,918][__main__][INFO] - Number of regex retries in iteration 370: 1
[2025-11-13 03:40:36,918][__main__][INFO] - agents played in iteration 370 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:40:37,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:40:37,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:40:37,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:40:37,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:40:37,857][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:40:37,858][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:40:38,554][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:40:39,012][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:40:39,516][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:40:40,016][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:40:40,515][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:40:41,013][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:40:41,512][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:40:42,009][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:40:42,506][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:40:43,005][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:40:43,505][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:40:44,004][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:40:44,507][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:40:45,006][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:40:45,505][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:40:46,013][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:40:46,516][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:40:47,016][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:40:47,520][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:40:48,020][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:40:48,521][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:40:49,022][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:40:49,528][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:40:50,031][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:40:50,542][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:40:51,040][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:40:51,560][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:40:52,064][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:40:52,563][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:40:53,063][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:40:53,562][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:40:54,060][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:40:54,559][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:40:55,058][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:40:55,572][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:40:56,070][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:40:56,570][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:40:57,067][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:40:57,565][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:40:58,074][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:40:58,574][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:40:59,073][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:40:59,573][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:41:00,072][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:41:00,572][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:41:01,073][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:41:01,568][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:41:02,074][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:41:02,572][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:41:03,071][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:41:03,570][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:41:04,069][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:41:04,575][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:41:05,073][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:41:05,572][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:41:06,086][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:41:06,585][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:41:07,084][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:41:07,582][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:41:08,082][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:41:08,585][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:41:09,086][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:41:09,585][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:41:10,089][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:41:10,585][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10142 tokens.
[2025-11-13 03:41:11,408][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 03:41:12,163][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:41:12,165][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:41:12,166][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:41:13,907][__main__][INFO] - Iteration 371 took 54s (32.63% Gen, 64.20% Train). Generation: 17s, Training: 35s. Estimated remaining time: 40h 13m 42s. Estimated total time: 45h 45m 14s. Time estimates for 10 more iterations: 9m 9s, 100 more iterations: 1h 31m 30s, 500 more iterations: 7h 37m 32s.
[2025-11-13 03:41:13,909][__main__][INFO] - Starting iteration 371.
[2025-11-13 03:41:14,397][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:41:14,397][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:41:29,200][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:41:30,378][__main__][INFO] - Number of regex retries in iteration 371: 1
[2025-11-13 03:41:30,379][__main__][INFO] - agents played in iteration 371 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:41:31,261][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:41:31,284][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:41:31,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:41:31,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:41:31,329][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:41:31,329][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:41:32,024][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:41:32,489][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:41:32,996][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:41:33,495][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:41:33,992][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:41:34,491][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:41:34,994][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:41:35,494][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:41:35,989][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:41:36,496][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:41:36,994][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:41:37,494][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:41:37,987][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:41:38,483][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:41:38,982][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:41:39,480][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:41:39,978][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:41:40,478][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:41:40,975][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:41:41,473][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:41:41,975][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:41:42,475][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:41:42,971][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:41:43,468][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:41:43,969][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:41:44,469][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:41:44,966][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:41:45,467][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:41:45,972][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:41:46,474][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:41:46,994][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:41:47,493][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:41:47,998][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:41:48,500][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:41:49,001][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:41:49,501][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:41:50,004][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:41:50,503][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:41:51,007][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:41:51,506][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:41:52,005][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:41:52,503][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:41:53,004][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:41:53,508][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:41:54,007][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:41:54,506][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:41:55,007][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:41:55,505][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:41:56,004][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:41:56,505][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:41:57,009][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:41:57,520][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:41:58,020][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:41:58,521][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:41:59,021][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:41:59,521][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:42:00,025][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:42:00,525][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:42:01,027][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:42:01,531][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:42:02,037][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:42:02,538][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:42:03,044][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:42:03,544][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:42:04,047][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10169 tokens.
[2025-11-13 03:42:04,833][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 03:42:05,604][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:42:05,608][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:42:05,611][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:42:06,692][__main__][INFO] - Iteration 372 took 52s (30.56% Gen, 67.37% Train). Generation: 15s, Training: 35s. Estimated remaining time: 38h 2m 24s. Estimated total time: 43h 34m 49s. Time estimates for 10 more iterations: 8m 42s, 100 more iterations: 1h 27m 9s, 500 more iterations: 7h 15m 48s.
[2025-11-13 03:42:06,695][__main__][INFO] - Starting iteration 372.
[2025-11-13 03:42:07,193][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:42:07,193][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:42:23,536][__main__][INFO] - Number of regex retries in iteration 372: 0
[2025-11-13 03:42:23,536][__main__][INFO] - agents played in iteration 372 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:42:24,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:42:24,339][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:42:24,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:42:24,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:42:24,386][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:42:24,387][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:42:25,084][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:42:25,536][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:42:26,038][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:42:26,537][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:42:27,038][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:42:27,540][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:42:28,038][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:42:28,535][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:42:29,038][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:42:29,536][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:42:30,031][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:42:30,529][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:42:31,026][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:42:31,532][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:42:32,028][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:42:32,527][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:42:33,026][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:42:33,522][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:42:34,019][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:42:34,517][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:42:35,018][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:42:35,521][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:42:36,020][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:42:36,522][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:42:37,024][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:42:37,529][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:42:38,033][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:42:38,561][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:42:39,064][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:42:39,575][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:42:40,077][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:42:40,579][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:42:41,082][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:42:41,578][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:42:42,080][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:42:42,580][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:42:43,082][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:42:43,596][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:42:44,094][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:42:44,598][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:42:45,095][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:42:45,598][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:42:46,103][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:42:46,603][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:42:47,105][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:42:47,608][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:42:48,112][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:42:48,614][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:42:49,113][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:42:49,612][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:42:50,115][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:42:50,615][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:42:51,116][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:42:51,617][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:42:52,118][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:42:52,618][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:42:53,121][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:42:53,625][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:42:54,133][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:42:54,638][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:42:55,142][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:42:55,645][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:42:56,147][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:42:56,655][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:42:57,157][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10275 tokens.
[2025-11-13 03:42:57,965][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:32
[2025-11-13 03:42:58,742][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:42:58,744][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:42:58,746][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:42:59,714][__main__][INFO] - Iteration 373 took 52s (31.12% Gen, 67.04% Train). Generation: 16s, Training: 35s. Estimated remaining time: 38h 12m 48s. Estimated total time: 43h 46m 6s. Time estimates for 10 more iterations: 8m 45s, 100 more iterations: 1h 27m 32s, 500 more iterations: 7h 17m 41s.
[2025-11-13 03:42:59,716][__main__][INFO] - Starting iteration 373.
[2025-11-13 03:43:00,195][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:43:00,196][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:43:18,131][__main__][INFO] - Number of regex retries in iteration 373: 0
[2025-11-13 03:43:18,132][__main__][INFO] - agents played in iteration 373 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:43:18,936][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:43:18,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:43:18,985][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:43:19,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:43:19,008][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:43:19,009][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:43:19,697][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:43:20,157][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:43:20,657][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:43:21,154][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:43:21,657][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:43:22,154][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:43:22,652][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:43:23,147][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:43:23,652][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:43:24,154][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:43:24,652][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:43:25,151][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:43:25,655][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:43:26,153][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:43:26,650][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:43:27,149][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:43:27,647][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:43:28,145][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:43:28,643][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:43:29,140][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:43:29,645][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:43:30,144][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:43:30,652][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:43:31,152][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:43:31,658][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:43:32,164][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:43:32,661][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:43:33,164][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:43:33,673][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:43:34,171][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:43:34,684][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:43:35,185][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:43:35,685][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:43:36,190][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:43:36,688][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:43:37,188][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:43:37,688][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:43:38,188][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:43:38,692][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:43:39,189][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:43:39,688][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:43:40,187][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:43:40,691][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:43:41,203][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:43:41,703][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:43:42,206][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:43:42,718][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:43:43,222][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:43:43,724][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:43:44,228][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:43:44,730][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:43:45,238][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:43:45,740][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:43:46,242][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:43:46,746][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:43:47,246][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:43:47,749][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:43:48,247][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:43:48,747][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:43:49,250][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:43:49,751][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:43:50,254][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:43:50,750][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:43:51,247][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:43:51,744][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10171 tokens.
[2025-11-13 03:43:52,494][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 03:43:53,306][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:43:53,308][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:43:53,310][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:43:54,508][__main__][INFO] - Iteration 374 took 54s (33.02% Gen, 64.77% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 41m 28s. Estimated total time: 45h 15m 41s. Time estimates for 10 more iterations: 9m 3s, 100 more iterations: 1h 30m 31s, 500 more iterations: 7h 32m 36s.
[2025-11-13 03:43:54,510][__main__][INFO] - Starting iteration 374.
[2025-11-13 03:43:54,995][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:43:54,995][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:44:13,094][__main__][INFO] - Number of regex retries in iteration 374: 0
[2025-11-13 03:44:13,095][__main__][INFO] - agents played in iteration 374 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:44:13,986][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:44:14,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:44:14,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:44:14,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:44:14,053][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:44:14,054][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:44:14,756][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:44:15,227][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:44:15,734][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:44:16,237][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:44:16,736][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:44:17,241][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:44:17,740][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:44:18,241][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:44:18,739][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:44:19,246][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:44:19,741][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:44:20,242][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:44:20,746][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:44:21,242][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:44:21,748][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:44:22,251][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:44:22,749][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:44:23,248][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:44:23,749][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:44:24,249][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:44:24,751][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:44:25,252][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:44:25,752][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:44:26,251][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:44:26,750][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:44:27,250][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:44:27,749][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:44:28,249][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:44:28,748][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:44:29,248][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:44:29,756][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:44:30,255][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:44:30,758][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:44:31,259][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:44:31,759][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:44:32,262][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:44:32,760][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:44:33,261][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:44:33,762][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:44:34,262][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:44:34,762][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:44:35,265][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:44:35,765][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:44:36,274][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:44:36,777][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:44:37,277][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:44:37,779][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:44:38,281][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:44:38,781][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:44:39,280][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:44:39,778][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:44:40,279][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:44:40,777][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:44:41,276][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:44:41,774][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:44:42,273][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:44:42,780][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:44:43,282][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:44:43,780][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:44:44,282][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:44:44,783][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:44:45,281][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:44:45,779][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:44:46,280][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:44:46,779][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10141 tokens.
[2025-11-13 03:44:47,548][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 03:44:48,335][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:44:48,336][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:44:48,338][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:44:49,353][__main__][INFO] - Iteration 375 took 54s (33.30% Gen, 64.83% Train). Generation: 18s, Training: 35s. Estimated remaining time: 39h 42m 49s. Estimated total time: 45h 17m 57s. Time estimates for 10 more iterations: 9m 3s, 100 more iterations: 1h 30m 35s, 500 more iterations: 7h 32m 59s.
[2025-11-13 03:44:49,355][__main__][INFO] - Starting iteration 375.
[2025-11-13 03:44:49,835][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:44:49,836][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:45:05,640][__main__][INFO] - Number of regex retries in iteration 375: 0
[2025-11-13 03:45:05,641][__main__][INFO] - agents played in iteration 375 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:45:06,488][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:45:06,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:45:06,546][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:45:06,583][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:45:06,584][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:45:06,584][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:45:07,295][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:45:07,751][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:45:08,259][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:45:08,761][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:45:09,264][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:45:09,760][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:45:10,259][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:45:10,757][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:45:11,255][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:45:11,752][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:45:12,248][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:45:12,749][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:45:13,252][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:45:13,748][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:45:14,248][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:45:14,757][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:45:15,259][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:45:15,758][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:45:16,259][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:45:16,758][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:45:17,271][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:45:17,772][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:45:18,273][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:45:18,771][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:45:19,276][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:45:19,783][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:45:20,280][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:45:20,778][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:45:21,278][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:45:21,776][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:45:22,275][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:45:22,775][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:45:23,272][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:45:23,779][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:45:24,278][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:45:24,778][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:45:25,285][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:45:25,783][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:45:26,284][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:45:26,784][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:45:27,283][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:45:27,787][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:45:28,286][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:45:28,786][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:45:29,286][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:45:29,784][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:45:30,282][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:45:30,779][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:45:31,278][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:45:31,800][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:45:32,301][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:45:32,800][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:45:33,310][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:45:33,812][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:45:34,321][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:45:34,822][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:45:35,324][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:45:35,832][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:45:36,335][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:45:36,846][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:45:37,346][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:45:37,848][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:45:38,348][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:45:38,845][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:45:39,343][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10108 tokens.
[2025-11-13 03:45:40,110][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 03:45:40,877][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:45:40,879][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:45:40,881][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:45:41,813][__main__][INFO] - Iteration 376 took 51s (30.40% Gen, 67.80% Train). Generation: 15s, Training: 35s. Estimated remaining time: 37h 42m 55s. Estimated total time: 43h 18m 56s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 37s, 500 more iterations: 7h 13m 9s.
[2025-11-13 03:45:41,815][__main__][INFO] - Starting iteration 376.
[2025-11-13 03:45:42,304][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:45:42,305][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:45:47,549][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:45:47,605][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:46:00,358][__main__][INFO] - Number of regex retries in iteration 376: 2
[2025-11-13 03:46:00,359][__main__][INFO] - agents played in iteration 376 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:46:01,189][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:46:01,217][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:46:01,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:46:01,266][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:46:01,266][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:46:01,267][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:46:01,966][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:46:02,420][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:46:02,923][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:46:03,419][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:46:03,916][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:46:04,419][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:46:04,914][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:46:05,408][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:46:05,910][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:46:06,407][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:46:06,904][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:46:07,409][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:46:07,905][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:46:08,402][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:46:08,898][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:46:09,395][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:46:09,899][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:46:10,397][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:46:10,895][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:46:11,397][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:46:11,894][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:46:12,392][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:46:12,891][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:46:13,393][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:46:13,894][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:46:14,392][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:46:14,893][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:46:15,412][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:46:15,911][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:46:16,411][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:46:16,908][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:46:17,413][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:46:17,920][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:46:18,423][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:46:18,930][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:46:19,432][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:46:19,935][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:46:20,438][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:46:20,938][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:46:21,438][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:46:21,938][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:46:22,442][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:46:22,959][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:46:23,460][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:46:23,965][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:46:24,472][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:46:24,973][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:46:25,478][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:46:25,980][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:46:26,480][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:46:26,979][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:46:27,475][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:46:27,978][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:46:28,476][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:46:28,976][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:46:29,477][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:46:29,979][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:46:30,479][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:46:30,990][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:46:31,490][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:46:31,990][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:46:32,493][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:46:32,993][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:46:33,500][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:46:34,000][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10140 tokens.
[2025-11-13 03:46:34,778][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 03:46:35,568][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:46:35,569][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:46:35,571][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:46:36,551][__main__][INFO] - Iteration 377 took 54s (33.28% Gen, 64.91% Train). Generation: 18s, Training: 35s. Estimated remaining time: 39h 35m 25s. Estimated total time: 45h 12m 19s. Time estimates for 10 more iterations: 9m 2s, 100 more iterations: 1h 30m 24s, 500 more iterations: 7h 32m 3s.
[2025-11-13 03:46:36,553][__main__][INFO] - Starting iteration 377.
[2025-11-13 03:46:37,037][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:46:37,037][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:46:56,091][__main__][INFO] - Number of regex retries in iteration 377: 0
[2025-11-13 03:46:56,092][__main__][INFO] - agents played in iteration 377 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:46:56,944][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:46:56,972][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:46:56,999][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:46:57,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:46:57,022][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:46:57,023][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:46:57,750][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:46:58,212][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:46:58,720][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:46:59,223][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:46:59,727][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:47:00,240][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:47:00,740][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:47:01,244][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:47:01,745][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:47:02,246][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:47:02,760][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:47:03,265][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:47:03,769][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:47:04,272][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:47:04,773][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:47:05,283][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:47:05,782][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:47:06,283][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:47:06,785][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:47:07,286][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:47:07,791][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:47:08,289][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:47:08,788][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:47:09,289][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:47:09,789][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:47:10,291][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:47:10,790][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:47:11,295][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:47:11,802][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:47:12,301][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:47:12,802][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:47:13,303][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:47:13,803][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:47:14,309][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:47:14,807][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:47:15,308][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:47:15,812][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:47:16,309][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:47:16,806][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:47:17,305][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:47:17,803][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:47:18,307][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:47:18,811][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:47:19,313][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:47:19,815][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:47:20,314][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:47:20,812][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:47:21,309][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:47:21,813][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:47:22,323][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:47:22,824][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:47:23,324][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:47:23,827][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:47:24,335][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:47:24,840][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:47:25,345][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:47:25,845][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:47:26,360][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:47:26,859][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:47:27,361][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:47:27,862][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:47:28,365][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:47:28,866][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:47:29,364][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:47:29,864][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10227 tokens.
[2025-11-13 03:47:30,728][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 03:47:31,515][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:47:31,517][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:47:31,519][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:47:32,493][__main__][INFO] - Iteration 378 took 55s (34.36% Gen, 63.88% Train). Generation: 19s, Training: 35s. Estimated remaining time: 40h 34m 57s. Estimated total time: 46h 12m 48s. Time estimates for 10 more iterations: 9m 14s, 100 more iterations: 1h 32m 25s, 500 more iterations: 7h 42m 8s.
[2025-11-13 03:47:32,495][__main__][INFO] - Starting iteration 378.
[2025-11-13 03:47:32,972][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:47:32,973][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:47:38,632][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:47:50,536][__main__][INFO] - Number of regex retries in iteration 378: 1
[2025-11-13 03:47:50,537][__main__][INFO] - agents played in iteration 378 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:47:51,441][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:47:51,466][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:47:51,490][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:47:51,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:47:51,513][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:47:51,513][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:47:52,216][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:47:52,676][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:47:53,182][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:47:53,684][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:47:54,195][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:47:54,697][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:47:55,200][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:47:55,700][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:47:56,200][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:47:56,701][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:47:57,197][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:47:57,695][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:47:58,205][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:47:58,706][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:47:59,206][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:47:59,709][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:48:00,208][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:48:00,708][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:48:01,206][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:48:01,704][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:48:02,204][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:48:02,703][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:48:03,203][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:48:03,704][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:48:04,205][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:48:04,720][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:48:05,226][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:48:05,729][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:48:06,238][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:48:06,742][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:48:07,247][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:48:07,746][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:48:08,250][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:48:08,754][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:48:09,253][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:48:09,752][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:48:10,255][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:48:10,758][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:48:11,261][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:48:11,761][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:48:12,261][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:48:12,771][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:48:13,270][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:48:13,777][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:48:14,278][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:48:14,780][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:48:15,286][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:48:15,789][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:48:16,286][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:48:16,803][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:48:17,301][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:48:17,801][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:48:18,304][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:48:18,801][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:48:19,304][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:48:19,806][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:48:20,305][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:48:20,804][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:48:21,302][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:48:21,808][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:48:22,309][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:48:22,816][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:48:23,322][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:48:23,824][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:48:24,324][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10262 tokens.
[2025-11-13 03:48:25,152][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 03:48:25,945][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:48:25,947][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:48:25,949][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:48:26,887][__main__][INFO] - Iteration 379 took 53s (32.58% Gen, 65.68% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 17m 0s. Estimated total time: 44h 55m 46s. Time estimates for 10 more iterations: 8m 59s, 100 more iterations: 1h 29m 51s, 500 more iterations: 7h 29m 17s.
[2025-11-13 03:48:26,889][__main__][INFO] - Starting iteration 379.
[2025-11-13 03:48:27,387][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:48:27,387][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:48:45,116][__main__][INFO] - Number of regex retries in iteration 379: 0
[2025-11-13 03:48:45,117][__main__][INFO] - agents played in iteration 379 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:48:45,968][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:48:45,990][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:48:46,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:48:46,034][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:48:46,035][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:48:46,035][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:48:46,748][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:48:47,201][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:48:47,707][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:48:48,215][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:48:48,711][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:48:49,213][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:48:49,715][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:48:50,213][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:48:50,710][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:48:51,206][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:48:51,701][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:48:52,220][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:48:52,724][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:48:53,223][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:48:53,720][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:48:54,224][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:48:54,728][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:48:55,229][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:48:55,730][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:48:56,237][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:48:56,738][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:48:57,236][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:48:57,734][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:48:58,235][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:48:58,738][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:48:59,240][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:48:59,741][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:49:00,242][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:49:00,747][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:49:01,248][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:49:01,747][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:49:02,246][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:49:02,754][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:49:03,264][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:49:03,873][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:49:04,397][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:49:04,904][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:49:05,435][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:49:05,939][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:49:06,444][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:49:06,946][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:49:07,447][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:49:07,951][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:49:08,452][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:49:08,956][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:49:09,461][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:49:09,960][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:49:10,463][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:49:10,962][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:49:11,466][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:49:11,977][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:49:12,477][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:49:12,985][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:49:13,488][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:49:13,990][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:49:14,493][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:49:14,995][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:49:15,499][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:49:16,007][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:49:16,507][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:49:17,021][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:49:17,520][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:49:18,025][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:49:18,524][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:49:19,025][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10279 tokens.
[2025-11-13 03:49:19,877][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:33
[2025-11-13 03:49:20,528][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:49:20,529][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:49:20,531][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:49:21,356][__main__][INFO] - Iteration 380 took 53s (32.85% Gen, 65.62% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 18m 50s. Estimated total time: 44h 58m 30s. Time estimates for 10 more iterations: 8m 59s, 100 more iterations: 1h 29m 57s, 500 more iterations: 7h 29m 45s.
[2025-11-13 03:49:21,358][__main__][INFO] - Starting iteration 380.
[2025-11-13 03:49:21,853][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 03:49:21,855][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:50:01,969][__main__][INFO] - Number of regex retries in iteration 380: 0
[2025-11-13 03:50:01,970][__main__][INFO] - agents played in iteration 380 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:50:02,892][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:50:02,915][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:50:02,939][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:50:02,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:50:02,962][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:50:02,963][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:50:03,767][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:50:04,238][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:50:04,746][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:50:05,254][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:50:05,760][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:50:06,264][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:50:06,769][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:50:07,271][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:50:07,774][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:50:08,278][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:50:08,782][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:50:09,285][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:50:09,788][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:50:10,290][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:50:10,799][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:50:11,299][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:50:11,801][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:50:12,314][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:50:12,820][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:50:13,343][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:50:13,850][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:50:14,354][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:50:14,859][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:50:15,365][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:50:15,872][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:50:16,373][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:50:16,874][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:50:17,376][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:50:17,877][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:50:18,377][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:50:18,879][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:50:19,379][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:50:19,883][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:50:20,383][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:50:20,886][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:50:21,390][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:50:21,890][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:50:22,399][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:50:22,901][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:50:23,405][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:50:23,914][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:50:24,416][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:50:24,928][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:50:25,431][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:50:25,930][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:50:26,440][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:50:26,938][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:50:27,440][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:50:27,938][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:50:28,438][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:50:28,949][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:50:29,455][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:50:29,958][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:50:30,460][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:50:30,962][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:50:31,465][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:50:31,969][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:50:32,471][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:50:32,972][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:50:33,473][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:50:33,972][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:50:34,474][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:50:34,976][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:50:35,477][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:50:35,973][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10145 tokens.
[2025-11-13 03:50:36,753][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.46%, ΔTime: 00:00:32
[2025-11-13 03:50:37,508][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:50:37,509][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:50:37,511][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:50:39,293][__main__][INFO] - Iteration 381 took 1m 17s (51.80% Gen, 45.89% Train). Generation: 40s, Training: 35s. Estimated remaining time: 58h 51m 5s. Estimated total time: 64h 32m 2s. Time estimates for 10 more iterations: 12m 54s, 100 more iterations: 2h 9m 4s, 500 more iterations: 10h 45m 20s.
[2025-11-13 03:50:39,295][__main__][INFO] - Starting iteration 381.
[2025-11-13 03:50:39,785][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:50:39,785][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:50:56,540][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:51:05,968][__main__][INFO] - Number of regex retries in iteration 381: 1
[2025-11-13 03:51:05,968][__main__][INFO] - agents played in iteration 381 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:51:06,834][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:51:06,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:51:06,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:51:06,910][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:51:06,910][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:51:06,912][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:51:07,708][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:51:08,165][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:51:08,677][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:51:09,180][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:51:09,683][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:51:10,187][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:51:10,687][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:51:11,192][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:51:11,694][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:51:12,198][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:51:12,699][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:51:13,200][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:51:13,716][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:51:14,217][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:51:14,716][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:51:15,218][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:51:15,719][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:51:16,243][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:51:16,743][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:51:17,245][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:51:17,745][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:51:18,245][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:51:18,752][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:51:19,250][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:51:19,750][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:51:20,252][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:51:20,753][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:51:21,257][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:51:21,757][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:51:22,257][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:51:22,766][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:51:23,267][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:51:23,768][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:51:24,270][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:51:24,770][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:51:25,272][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:51:25,771][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:51:26,271][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:51:26,776][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:51:27,274][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:51:27,778][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:51:28,277][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:51:28,778][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:51:29,281][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:51:29,783][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:51:30,281][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:51:30,783][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:51:31,280][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:51:31,776][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:51:32,273][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:51:32,769][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:51:33,266][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:51:33,764][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:51:34,261][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:51:34,756][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:51:35,251][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:51:35,752][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:51:36,247][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:51:36,746][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:51:37,261][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:51:37,764][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:51:38,270][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:51:38,771][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:51:39,275][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:51:39,775][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10165 tokens.
[2025-11-13 03:51:40,502][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 03:51:41,268][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:51:41,269][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:51:41,271][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:51:42,381][__main__][INFO] - Iteration 382 took 1m 2s (41.83% Gen, 56.40% Train). Generation: 26s, Training: 35s. Estimated remaining time: 46h 27m 48s. Estimated total time: 52h 9m 49s. Time estimates for 10 more iterations: 10m 25s, 100 more iterations: 1h 44m 19s, 500 more iterations: 8h 41m 38s.
[2025-11-13 03:51:42,383][__main__][INFO] - Starting iteration 382.
[2025-11-13 03:51:42,851][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:51:42,852][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:52:00,974][__main__][INFO] - Number of regex retries in iteration 382: 0
[2025-11-13 03:52:00,975][__main__][INFO] - agents played in iteration 382 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:52:01,835][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:52:01,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:52:01,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:52:01,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:52:01,907][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:52:01,908][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:52:02,701][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:52:03,160][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:52:03,675][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:52:04,176][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:52:04,677][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:52:05,178][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:52:05,683][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:52:06,186][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:52:06,689][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:52:07,195][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:52:07,702][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:52:08,202][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:52:08,705][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:52:09,220][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:52:09,724][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:52:10,239][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:52:10,742][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:52:11,245][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:52:11,754][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:52:12,258][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:52:12,762][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:52:13,263][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:52:13,764][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:52:14,269][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:52:14,774][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:52:15,277][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:52:15,782][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:52:16,281][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:52:16,784][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:52:17,284][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:52:17,784][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:52:18,285][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:52:18,788][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:52:19,288][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:52:19,789][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:52:20,287][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:52:20,789][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:52:21,286][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:52:21,784][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:52:22,283][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:52:22,781][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:52:23,283][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:52:23,786][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:52:24,285][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:52:24,790][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:52:25,293][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:52:25,794][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:52:26,293][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:52:26,796][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:52:27,295][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:52:27,790][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:52:28,286][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:52:28,784][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:52:29,281][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:52:29,778][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:52:30,274][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:52:30,770][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:52:31,271][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:52:31,770][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:52:32,273][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:52:32,768][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:52:33,262][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:52:33,762][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:52:34,257][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:52:34,756][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10153 tokens.
[2025-11-13 03:52:35,501][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:32
[2025-11-13 03:52:36,267][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:52:36,270][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:52:36,272][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:52:37,436][__main__][INFO] - Iteration 383 took 54s (33.20% Gen, 64.67% Train). Generation: 18s, Training: 35s. Estimated remaining time: 39h 46m 18s. Estimated total time: 45h 29m 14s. Time estimates for 10 more iterations: 9m 5s, 100 more iterations: 1h 30m 58s, 500 more iterations: 7h 34m 52s.
[2025-11-13 03:52:37,438][__main__][INFO] - Starting iteration 383.
[2025-11-13 03:52:37,925][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:52:37,926][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:52:55,657][__main__][INFO] - Number of regex retries in iteration 383: 0
[2025-11-13 03:52:55,657][__main__][INFO] - agents played in iteration 383 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:52:56,446][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:52:56,475][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:52:56,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:52:56,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:52:56,525][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:52:56,525][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:52:57,246][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:52:57,701][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:52:58,207][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:52:58,712][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:52:59,217][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:52:59,720][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:53:00,223][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:53:00,754][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:53:01,267][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:53:01,769][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:53:02,270][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:53:02,770][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:53:03,272][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:53:03,786][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:53:04,285][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:53:04,788][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:53:05,288][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:53:05,788][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:53:06,290][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:53:06,794][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:53:07,293][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:53:07,796][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:53:08,305][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:53:08,807][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:53:09,314][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:53:09,815][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:53:10,318][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:53:10,821][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:53:11,320][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:53:11,821][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:53:12,322][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:53:12,824][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:53:13,324][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:53:13,827][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:53:14,338][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:53:14,838][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:53:15,343][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:53:15,845][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:53:16,347][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:53:16,848][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:53:17,349][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:53:17,848][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:53:18,354][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:53:18,852][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:53:19,370][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:53:19,868][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:53:20,367][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:53:20,864][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:53:21,363][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:53:21,873][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:53:22,377][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:53:22,877][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:53:23,377][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:53:23,875][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:53:24,372][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:53:24,872][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:53:25,373][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:53:25,882][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:53:26,380][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:53:26,883][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:53:27,380][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:53:27,904][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:53:28,406][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:53:28,909][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:53:29,416][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10100 tokens.
[2025-11-13 03:53:30,153][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 03:53:30,927][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:53:30,929][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:53:30,931][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:53:31,934][__main__][INFO] - Iteration 384 took 54s (32.83% Gen, 65.31% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 16m 38s. Estimated total time: 45h 0m 28s. Time estimates for 10 more iterations: 9m 0s, 100 more iterations: 1h 30m 0s, 500 more iterations: 7h 30m 4s.
[2025-11-13 03:53:31,936][__main__][INFO] - Starting iteration 384.
[2025-11-13 03:53:32,421][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:53:32,421][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:53:51,131][__main__][INFO] - Number of regex retries in iteration 384: 0
[2025-11-13 03:53:51,132][__main__][INFO] - agents played in iteration 384 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:53:51,962][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:53:51,984][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:53:52,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:53:52,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:53:52,030][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:53:52,030][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:53:52,794][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:53:53,268][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:53:53,777][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:53:54,275][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:53:54,786][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:53:55,287][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:53:55,788][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:53:56,291][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:53:56,791][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:53:57,293][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:53:57,800][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:53:58,301][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:53:58,808][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:53:59,315][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:53:59,814][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:54:00,318][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:54:00,819][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:54:01,321][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:54:01,824][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:54:02,325][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:54:02,828][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:54:03,326][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:54:03,828][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:54:04,330][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:54:04,831][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:54:05,340][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:54:05,840][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:54:06,339][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:54:06,851][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:54:07,351][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:54:07,850][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:54:08,352][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:54:08,853][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:54:09,364][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:54:09,865][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:54:10,363][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:54:10,870][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:54:11,367][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:54:11,882][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:54:12,378][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:54:12,879][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:54:13,388][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:54:13,886][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:54:14,384][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:54:14,882][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:54:15,380][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:54:15,887][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:54:16,384][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:54:16,881][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:54:17,385][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:54:17,882][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:54:18,383][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:54:18,883][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:54:19,386][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:54:19,893][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:54:20,392][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:54:20,889][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:54:21,387][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:54:21,890][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:54:22,397][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:54:22,895][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:54:23,395][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:54:23,898][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:54:24,400][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:54:24,897][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10125 tokens.
[2025-11-13 03:54:25,641][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.02%, ΔTime: 00:00:32
[2025-11-13 03:54:26,419][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:54:26,420][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:54:26,422][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:54:27,603][__main__][INFO] - Iteration 385 took 55s (33.91% Gen, 63.95% Train). Generation: 18s, Training: 35s. Estimated remaining time: 40h 14m 21s. Estimated total time: 45h 59m 7s. Time estimates for 10 more iterations: 9m 11s, 100 more iterations: 1h 31m 58s, 500 more iterations: 7h 39m 51s.
[2025-11-13 03:54:27,605][__main__][INFO] - Starting iteration 385.
[2025-11-13 03:54:28,114][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:54:28,115][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:54:39,856][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:54:45,089][__main__][INFO] - Number of regex retries in iteration 385: 1
[2025-11-13 03:54:45,090][__main__][INFO] - agents played in iteration 385 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:54:45,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:54:45,901][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:54:45,925][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:54:45,948][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:54:45,948][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:54:45,949][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:54:46,735][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:54:47,195][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:54:47,698][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:54:48,197][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:54:48,709][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:54:49,207][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:54:49,707][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:54:50,203][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:54:50,700][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:54:51,201][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:54:51,705][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:54:52,202][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:54:52,699][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:54:53,197][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:54:53,696][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:54:54,199][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:54:54,698][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:54:55,208][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:54:55,708][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:54:56,218][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:54:56,719][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:54:57,221][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:54:57,725][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:54:58,226][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:54:58,728][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:54:59,230][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:54:59,736][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:55:00,240][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:55:00,740][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:55:01,244][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:55:01,748][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:55:02,247][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:55:02,748][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:55:03,246][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:55:03,744][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:55:04,242][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:55:04,739][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:55:05,239][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:55:05,744][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:55:06,245][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:55:06,741][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:55:07,241][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:55:07,738][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:55:08,236][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:55:08,735][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:55:09,232][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:55:09,731][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:55:10,231][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:55:10,729][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:55:11,225][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:55:11,723][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:55:12,224][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:55:12,723][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:55:13,224][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:55:13,724][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:55:14,226][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:55:14,726][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:55:15,224][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:55:15,747][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:55:16,248][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:55:16,747][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:55:17,247][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:55:17,746][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:55:18,243][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:55:18,742][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10055 tokens.
[2025-11-13 03:55:19,459][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:32
[2025-11-13 03:55:20,215][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:55:20,217][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:55:20,218][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:55:21,190][__main__][INFO] - Iteration 386 took 53s (31.98% Gen, 66.18% Train). Generation: 16s, Training: 35s. Estimated remaining time: 38h 28m 9s. Estimated total time: 44h 13m 48s. Time estimates for 10 more iterations: 8m 50s, 100 more iterations: 1h 28m 27s, 500 more iterations: 7h 22m 18s.
[2025-11-13 03:55:21,192][__main__][INFO] - Starting iteration 386.
[2025-11-13 03:55:21,687][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:55:21,688][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:55:39,033][__main__][INFO] - Number of regex retries in iteration 386: 0
[2025-11-13 03:55:39,033][__main__][INFO] - agents played in iteration 386 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:55:39,841][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:55:39,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:55:39,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:55:39,909][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:55:39,910][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:55:39,910][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:55:40,659][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:55:41,121][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:55:41,622][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:55:42,126][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:55:42,624][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:55:43,120][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:55:43,621][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:55:44,124][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:55:44,621][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:55:45,122][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:55:45,623][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:55:46,136][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:55:46,653][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:55:47,162][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:55:47,674][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:55:48,175][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:55:48,684][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:55:49,183][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:55:49,682][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:55:50,197][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:55:50,697][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:55:51,197][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:55:51,707][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:55:52,208][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:55:52,719][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:55:53,220][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:55:53,719][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:55:54,236][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:55:54,737][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:55:55,236][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:55:55,737][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:55:56,237][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:55:56,739][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:55:57,238][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:55:57,738][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:55:58,245][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:55:58,743][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:55:59,242][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:55:59,744][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:56:00,243][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:56:00,743][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:56:01,242][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:56:01,742][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:56:02,242][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:56:02,743][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:56:03,243][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:56:03,742][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:56:04,246][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:56:04,750][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:56:05,254][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:56:05,754][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:56:06,256][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:56:06,760][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:56:07,265][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:56:07,766][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:56:08,267][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:56:08,767][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:56:09,265][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:56:09,763][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:56:10,262][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:56:10,762][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:56:11,265][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:56:11,766][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:56:12,263][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:56:12,763][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10095 tokens.
[2025-11-13 03:56:13,490][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 03:56:14,257][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:56:14,258][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:56:14,260][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:56:15,457][__main__][INFO] - Iteration 387 took 53s (32.26% Gen, 65.51% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 1m 55s. Estimated total time: 44h 48m 29s. Time estimates for 10 more iterations: 8m 57s, 100 more iterations: 1h 29m 36s, 500 more iterations: 7h 28m 4s.
[2025-11-13 03:56:15,459][__main__][INFO] - Starting iteration 387.
[2025-11-13 03:56:15,951][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:56:15,952][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:56:21,664][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:56:21,744][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:56:34,258][__main__][INFO] - Number of regex retries in iteration 387: 2
[2025-11-13 03:56:34,258][__main__][INFO] - agents played in iteration 387 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:56:35,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:56:35,118][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:56:35,144][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:56:35,166][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:56:35,166][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:56:35,167][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:56:35,899][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:56:36,354][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:56:36,862][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:56:37,366][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:56:37,871][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:56:38,370][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:56:38,877][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:56:39,377][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:56:39,877][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:56:40,380][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:56:40,881][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:56:41,389][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:56:41,888][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:56:42,389][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:56:42,893][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:56:43,393][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:56:43,895][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:56:44,399][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:56:44,901][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:56:45,402][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:56:45,902][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:56:46,406][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:56:46,903][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:56:47,402][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:56:47,923][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:56:48,424][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:56:48,924][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:56:49,434][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:56:49,936][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:56:50,447][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:56:50,950][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:56:51,448][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:56:51,949][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:56:52,447][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:56:52,945][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:56:53,443][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:56:53,943][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:56:54,455][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:56:54,953][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:56:55,457][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:56:55,958][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:56:56,458][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:56:56,960][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:56:57,458][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:56:57,957][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:56:58,460][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:56:58,958][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:56:59,456][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:56:59,953][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:57:00,450][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:57:00,962][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:57:01,459][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:57:01,957][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:57:02,455][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:57:02,959][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:57:03,458][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:57:03,959][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:57:04,460][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:57:04,961][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:57:05,459][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:57:05,959][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:57:06,457][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:57:06,955][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:57:07,457][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:57:07,954][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10117 tokens.
[2025-11-13 03:57:08,689][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.06%, ΔTime: 00:00:32
[2025-11-13 03:57:09,437][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:57:09,439][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:57:09,441][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:57:10,437][__main__][INFO] - Iteration 388 took 54s (33.60% Gen, 64.57% Train). Generation: 18s, Training: 35s. Estimated remaining time: 39h 36m 50s. Estimated total time: 45h 24m 19s. Time estimates for 10 more iterations: 9m 4s, 100 more iterations: 1h 30m 48s, 500 more iterations: 7h 34m 3s.
[2025-11-13 03:57:10,439][__main__][INFO] - Starting iteration 388.
[2025-11-13 03:57:10,920][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:57:10,921][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:57:27,777][__main__][INFO] - Number of regex retries in iteration 388: 0
[2025-11-13 03:57:27,777][__main__][INFO] - agents played in iteration 388 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:57:28,643][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:57:28,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:57:28,699][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:57:28,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:57:28,723][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:57:28,723][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:57:29,452][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:57:29,918][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:57:30,428][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:57:30,944][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:57:31,448][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:57:31,949][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:57:32,464][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:57:32,966][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:57:33,470][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:57:33,969][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:57:34,465][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:57:34,968][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:57:35,466][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:57:35,965][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:57:36,470][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:57:36,969][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:57:37,471][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:57:37,971][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:57:38,480][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:57:38,986][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:57:39,487][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:57:39,989][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:57:40,492][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:57:40,992][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:57:41,493][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:57:41,993][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:57:42,492][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:57:42,994][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:57:43,494][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:57:43,995][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:57:44,493][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:57:44,990][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:57:45,489][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:57:45,991][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:57:46,490][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:57:46,989][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:57:47,486][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:57:47,987][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:57:48,484][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:57:48,986][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:57:49,489][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:57:49,991][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:57:50,493][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:57:50,992][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:57:51,491][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:57:51,994][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:57:52,496][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:57:52,997][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:57:53,497][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:57:53,998][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:57:54,499][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:57:55,000][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:57:55,499][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:57:55,999][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:57:56,498][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:57:57,004][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:57:57,506][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:57:58,002][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:57:58,499][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:57:58,999][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:57:59,498][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:57:59,997][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:58:00,496][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:58:00,993][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:58:01,493][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10192 tokens.
[2025-11-13 03:58:02,248][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 03:58:02,985][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:58:02,986][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:58:02,988][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:58:03,904][__main__][INFO] - Iteration 389 took 52s (31.81% Gen, 66.46% Train). Generation: 16s, Training: 35s. Estimated remaining time: 38h 20m 48s. Estimated total time: 44h 9m 11s. Time estimates for 10 more iterations: 8m 49s, 100 more iterations: 1h 28m 18s, 500 more iterations: 7h 21m 31s.
[2025-11-13 03:58:03,906][__main__][INFO] - Starting iteration 389.
[2025-11-13 03:58:04,378][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:58:04,378][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:58:13,635][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:58:20,195][__main__][INFO] - Number of regex retries in iteration 389: 1
[2025-11-13 03:58:20,196][__main__][INFO] - agents played in iteration 389 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:58:21,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:58:21,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:58:21,087][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:58:21,109][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:58:21,110][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:58:21,110][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:58:21,867][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:58:22,325][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:58:22,829][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:58:23,334][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:58:23,833][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:58:24,338][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:58:24,840][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:58:25,343][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:58:25,868][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:58:26,370][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:58:26,875][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:58:27,376][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:58:27,880][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:58:28,381][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:58:28,883][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:58:29,384][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:58:29,885][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:58:30,386][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:58:30,889][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:58:31,388][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:58:31,887][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:58:32,389][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:58:32,887][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:58:33,389][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:58:33,889][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:58:34,387][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:58:34,890][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:58:35,388][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:58:35,890][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:58:36,394][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:58:36,899][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:58:37,400][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:58:37,902][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:58:38,404][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:58:38,905][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:58:39,405][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:58:39,902][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:58:40,403][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:58:40,904][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:58:41,405][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:58:41,903][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:58:42,400][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:58:42,898][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:58:43,394][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:58:43,896][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:58:44,396][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:58:44,896][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:58:45,393][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:58:45,897][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:58:46,398][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:58:46,898][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:58:47,401][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:58:47,904][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:58:48,403][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:58:48,902][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:58:49,402][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:58:49,904][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:58:50,404][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:58:50,901][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:58:51,399][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:58:51,896][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:58:52,392][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:58:52,888][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:58:53,388][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:58:53,891][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10167 tokens.
[2025-11-13 03:58:54,637][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 03:58:55,379][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:58:55,381][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:58:55,383][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:58:56,324][__main__][INFO] - Iteration 390 took 51s (30.45% Gen, 67.74% Train). Generation: 15s, Training: 35s. Estimated remaining time: 37h 28m 5s. Estimated total time: 43h 17m 20s. Time estimates for 10 more iterations: 8m 39s, 100 more iterations: 1h 26m 34s, 500 more iterations: 7h 12m 53s.
[2025-11-13 03:58:56,326][__main__][INFO] - Starting iteration 390.
[2025-11-13 03:58:56,797][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 03:58:56,797][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 03:59:01,536][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 03:59:13,420][__main__][INFO] - Number of regex retries in iteration 390: 1
[2025-11-13 03:59:13,420][__main__][INFO] - agents played in iteration 390 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 03:59:14,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:59:14,235][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:59:14,261][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:59:14,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 03:59:14,284][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 03:59:14,285][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 03:59:15,004][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 03:59:15,465][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 03:59:15,969][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 03:59:16,464][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 03:59:16,961][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 03:59:17,462][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 03:59:17,960][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 03:59:18,460][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 03:59:18,956][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 03:59:19,481][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 03:59:19,984][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 03:59:20,485][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 03:59:20,983][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 03:59:21,481][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 03:59:21,986][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 03:59:22,493][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 03:59:22,991][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 03:59:23,491][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 03:59:23,989][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 03:59:24,486][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 03:59:24,986][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 03:59:25,483][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 03:59:25,987][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 03:59:26,490][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 03:59:26,991][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 03:59:27,489][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 03:59:27,997][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 03:59:28,499][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 03:59:29,010][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 03:59:29,521][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 03:59:30,031][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 03:59:30,545][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 03:59:31,051][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 03:59:31,552][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 03:59:32,056][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 03:59:32,570][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 03:59:33,077][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 03:59:33,581][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 03:59:34,077][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 03:59:34,580][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 03:59:35,085][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 03:59:35,581][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 03:59:36,078][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 03:59:36,577][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 03:59:37,074][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 03:59:37,571][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 03:59:38,070][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 03:59:38,566][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 03:59:39,079][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 03:59:39,579][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 03:59:40,080][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 03:59:40,580][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 03:59:41,080][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 03:59:41,582][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 03:59:42,102][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 03:59:42,605][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 03:59:43,104][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 03:59:43,603][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 03:59:44,102][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 03:59:44,598][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 03:59:45,096][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 03:59:45,601][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 03:59:46,099][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 03:59:46,600][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 03:59:47,103][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10185 tokens.
[2025-11-13 03:59:47,844][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 03:59:48,603][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 03:59:48,604][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 03:59:48,606][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 03:59:50,966][__main__][INFO] - Iteration 391 took 54s (30.69% Gen, 64.96% Train). Generation: 16s, Training: 35s. Estimated remaining time: 39h 18m 18s. Estimated total time: 45h 8m 27s. Time estimates for 10 more iterations: 9m 1s, 100 more iterations: 1h 30m 16s, 500 more iterations: 7h 31m 24s.
[2025-11-13 03:59:50,968][__main__][INFO] - Starting iteration 391.
[2025-11-13 03:59:51,445][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 03:59:51,446][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:00:07,606][__main__][INFO] - Number of regex retries in iteration 391: 0
[2025-11-13 04:00:07,606][__main__][INFO] - agents played in iteration 391 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:00:08,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:00:08,408][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:00:08,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:00:08,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:00:08,455][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:00:08,456][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:00:09,198][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:00:09,652][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:00:10,157][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:00:10,656][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:00:11,156][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:00:11,670][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:00:12,170][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:00:12,669][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:00:13,178][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:00:13,675][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:00:14,185][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:00:14,683][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:00:15,184][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:00:15,686][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:00:16,184][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:00:16,682][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:00:17,181][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:00:17,678][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:00:18,194][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:00:18,700][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:00:19,199][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:00:19,700][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:00:20,203][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:00:20,706][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:00:21,206][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:00:21,707][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:00:22,205][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:00:22,702][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:00:23,202][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:00:23,705][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:00:24,214][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:00:24,719][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:00:25,224][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:00:25,723][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:00:26,226][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:00:26,727][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:00:27,226][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:00:27,732][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:00:28,230][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:00:28,731][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:00:29,235][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:00:29,733][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:00:30,231][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:00:30,728][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:00:31,225][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:00:31,725][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:00:32,221][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:00:32,722][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:00:33,218][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:00:33,715][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:00:34,217][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:00:34,714][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:00:35,214][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:00:35,709][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:00:36,205][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:00:36,704][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:00:37,204][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:00:37,704][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:00:38,211][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:00:38,711][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:00:39,217][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:00:39,717][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:00:40,220][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:00:40,721][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:00:41,219][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10140 tokens.
[2025-11-13 04:00:41,974][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 04:00:42,735][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:00:42,737][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:00:42,739][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:00:43,845][__main__][INFO] - Iteration 392 took 52s (30.84% Gen, 67.05% Train). Generation: 16s, Training: 35s. Estimated remaining time: 37h 49m 0s. Estimated total time: 43h 40m 2s. Time estimates for 10 more iterations: 8m 44s, 100 more iterations: 1h 27m 20s, 500 more iterations: 7h 16m 40s.
[2025-11-13 04:00:43,848][__main__][INFO] - Starting iteration 392.
[2025-11-13 04:00:44,338][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 04:00:44,339][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:01:00,797][__main__][INFO] - Number of regex retries in iteration 392: 0
[2025-11-13 04:01:00,797][__main__][INFO] - agents played in iteration 392 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:01:01,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:01:01,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:01:01,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:01:01,694][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:01:01,695][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:01:01,696][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:01:02,416][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:01:02,882][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:01:03,388][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:01:03,892][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:01:04,392][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:01:04,897][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:01:05,400][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:01:05,901][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:01:06,402][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:01:06,912][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:01:07,417][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:01:07,918][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:01:08,443][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:01:08,942][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:01:09,446][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:01:09,949][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:01:10,454][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:01:10,962][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:01:11,460][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:01:11,961][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:01:12,459][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:01:12,957][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:01:13,465][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:01:13,967][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:01:14,466][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:01:14,975][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:01:15,477][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:01:15,989][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:01:16,495][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:01:16,998][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:01:17,503][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:01:18,007][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:01:18,516][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:01:19,017][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:01:19,521][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:01:20,025][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:01:20,525][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:01:21,028][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:01:21,531][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:01:22,033][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:01:22,536][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:01:23,041][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:01:23,544][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:01:24,049][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:01:24,548][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:01:25,054][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:01:25,557][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:01:26,067][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:01:26,581][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:01:27,084][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:01:27,584][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:01:28,090][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:01:28,588][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:01:29,110][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:01:29,611][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:01:30,120][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:01:30,622][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:01:31,129][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:01:31,638][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:01:32,136][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:01:32,634][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:01:33,136][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:01:33,638][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:01:34,142][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:01:34,640][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10234 tokens.
[2025-11-13 04:01:35,418][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.47%, ΔTime: 00:00:33
[2025-11-13 04:01:36,202][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:01:36,203][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:01:36,206][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:01:37,169][__main__][INFO] - Iteration 393 took 52s (31.15% Gen, 67.02% Train). Generation: 16s, Training: 35s. Estimated remaining time: 38h 9m 38s. Estimated total time: 44h 1m 34s. Time estimates for 10 more iterations: 8m 48s, 100 more iterations: 1h 28m 3s, 500 more iterations: 7h 20m 15s.
[2025-11-13 04:01:37,171][__main__][INFO] - Starting iteration 393.
[2025-11-13 04:01:37,658][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 04:01:37,658][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:01:55,838][__main__][INFO] - Number of regex retries in iteration 393: 0
[2025-11-13 04:01:55,839][__main__][INFO] - agents played in iteration 393 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:01:56,742][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:01:56,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:01:56,793][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:01:56,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:01:56,817][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:01:56,818][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:01:57,620][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:01:58,079][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:01:58,580][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:01:59,080][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:01:59,581][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:02:00,078][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:02:00,595][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:02:01,092][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:02:01,588][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:02:02,090][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:02:02,584][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:02:03,089][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:02:03,585][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:02:04,081][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:02:04,582][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:02:05,084][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:02:05,587][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:02:06,084][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:02:06,582][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:02:07,085][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:02:07,584][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:02:08,083][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:02:08,583][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:02:09,083][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:02:09,600][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:02:10,101][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:02:10,600][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:02:11,102][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:02:11,602][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:02:12,111][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:02:12,609][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:02:13,108][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:02:13,620][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:02:14,121][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:02:14,623][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:02:15,122][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:02:15,622][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:02:16,125][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:02:16,630][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:02:17,133][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:02:17,635][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:02:18,132][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:02:18,633][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:02:19,131][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:02:19,629][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:02:20,131][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:02:20,628][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:02:21,126][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:02:21,625][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:02:22,123][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:02:22,628][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:02:23,126][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:02:23,626][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:02:24,131][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:02:24,632][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:02:25,130][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:02:25,629][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:02:26,126][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:02:26,638][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:02:27,134][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:02:27,629][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:02:28,129][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:02:28,626][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:02:29,131][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:02:29,630][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10155 tokens.
[2025-11-13 04:02:30,415][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 04:02:31,159][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:02:31,161][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:02:31,162][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:02:32,110][__main__][INFO] - Iteration 394 took 54s (33.39% Gen, 64.87% Train). Generation: 18s, Training: 35s. Estimated remaining time: 39h 29m 47s. Estimated total time: 45h 22m 37s. Time estimates for 10 more iterations: 9m 4s, 100 more iterations: 1h 30m 45s, 500 more iterations: 7h 33m 46s.
[2025-11-13 04:02:32,112][__main__][INFO] - Starting iteration 394.
[2025-11-13 04:02:32,652][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 04:02:32,653][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:02:49,996][__main__][INFO] - Number of regex retries in iteration 394: 0
[2025-11-13 04:02:49,997][__main__][INFO] - agents played in iteration 394 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:02:50,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:02:50,861][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:02:50,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:02:50,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:02:50,911][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:02:50,912][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:02:51,620][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:02:52,073][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:02:52,574][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:02:53,077][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:02:53,577][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:02:54,080][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:02:54,576][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:02:55,074][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:02:55,573][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:02:56,070][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:02:56,571][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:02:57,070][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:02:57,569][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:02:58,068][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:02:58,567][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:02:59,064][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:02:59,566][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:03:00,068][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:03:00,566][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:03:01,064][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:03:01,561][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:03:02,064][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:03:02,567][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:03:03,067][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:03:03,573][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:03:04,072][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:03:04,575][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:03:05,076][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:03:05,578][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:03:06,080][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:03:06,581][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:03:07,079][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:03:07,580][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:03:08,079][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:03:08,579][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:03:09,077][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:03:09,575][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:03:10,075][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:03:10,572][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:03:11,069][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:03:11,571][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:03:12,065][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:03:12,562][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:03:13,062][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:03:13,559][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:03:14,059][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:03:14,556][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:03:15,056][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:03:15,556][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:03:16,057][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:03:16,557][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:03:17,054][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:03:17,553][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:03:18,052][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:03:18,553][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:03:19,052][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:03:19,550][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:03:20,047][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:03:20,549][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:03:21,045][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:03:21,541][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:03:22,041][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:03:22,540][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:03:23,039][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:03:23,536][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10101 tokens.
[2025-11-13 04:03:24,313][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.01%, Current % of VRAM taken: 60.27%, Block Peak % of device VRAM: 62.13%, ΔTime: 00:00:32
[2025-11-13 04:03:25,075][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:03:25,079][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:03:25,081][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:03:26,073][__main__][INFO] - Iteration 395 took 53s (32.47% Gen, 65.68% Train). Generation: 17s, Training: 35s. Estimated remaining time: 38h 37m 19s. Estimated total time: 44h 31m 4s. Time estimates for 10 more iterations: 8m 54s, 100 more iterations: 1h 29m 2s, 500 more iterations: 7h 25m 10s.
[2025-11-13 04:03:26,075][__main__][INFO] - Starting iteration 395.
[2025-11-13 04:03:26,544][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 04:03:26,544][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:03:41,903][__main__][INFO] - Number of regex retries in iteration 395: 0
[2025-11-13 04:03:41,904][__main__][INFO] - agents played in iteration 395 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:03:42,733][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:03:42,755][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:03:42,778][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:03:42,800][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:03:42,800][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:03:42,801][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:03:43,508][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:03:43,965][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:03:44,469][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:03:44,970][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:03:45,468][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:03:45,968][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:03:46,470][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:03:46,972][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:03:47,470][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:03:47,973][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:03:48,472][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:03:48,970][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:03:49,473][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:03:49,976][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:03:50,480][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:03:50,984][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:03:51,491][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:03:51,994][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:03:52,495][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:03:53,009][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:03:53,518][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:03:54,019][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:03:54,518][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:03:55,026][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:03:55,527][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:03:56,041][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:03:56,546][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:03:57,049][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:03:57,551][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:03:58,055][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:03:58,560][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:03:59,061][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:03:59,562][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:04:00,069][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:04:00,574][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:04:01,076][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:04:01,580][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:04:02,084][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:04:02,585][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:04:03,085][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:04:03,585][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:04:04,083][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:04:04,585][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:04:05,082][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:04:05,583][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:04:06,081][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:04:06,581][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:04:07,079][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:04:07,577][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:04:08,080][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:04:08,578][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:04:09,077][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:04:09,575][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:04:10,077][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:04:10,579][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:04:11,080][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:04:11,578][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:04:12,077][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:04:12,579][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:04:13,082][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:04:13,583][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:04:14,084][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:04:14,591][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:04:15,090][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:04:15,592][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10150 tokens.
[2025-11-13 04:04:16,422][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 04:04:17,191][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:04:17,192][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:04:17,194][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:04:18,108][__main__][INFO] - Iteration 396 took 51s (29.79% Gen, 68.44% Train). Generation: 15s, Training: 35s. Estimated remaining time: 37h 3m 39s. Estimated total time: 42h 58m 15s. Time estimates for 10 more iterations: 8m 35s, 100 more iterations: 1h 25m 56s, 500 more iterations: 7h 9m 42s.
[2025-11-13 04:04:18,111][__main__][INFO] - Starting iteration 396.
[2025-11-13 04:04:18,597][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 04:04:18,597][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:04:36,976][__main__][INFO] - Number of regex retries in iteration 396: 0
[2025-11-13 04:04:36,977][__main__][INFO] - agents played in iteration 396 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:04:37,824][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:04:37,851][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:04:37,888][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:04:37,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:04:37,914][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:04:37,915][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:04:38,632][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:04:39,085][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:04:39,588][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:04:40,083][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:04:40,584][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:04:41,079][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:04:41,573][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:04:42,069][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:04:42,565][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:04:43,065][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:04:43,564][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:04:44,065][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:04:44,575][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:04:45,073][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:04:45,574][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:04:46,094][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:04:46,608][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:04:47,112][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:04:47,613][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:04:48,111][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:04:48,621][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:04:49,119][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:04:49,626][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:04:50,277][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:04:50,778][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:04:51,281][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:04:51,783][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:04:52,283][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:04:52,783][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:04:53,283][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:04:53,786][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:04:54,288][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:04:54,794][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:04:55,292][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:04:55,794][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:04:56,302][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:04:56,801][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:04:57,299][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:04:57,805][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:04:58,306][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:04:58,805][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:04:59,310][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:04:59,815][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:05:00,317][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:05:00,818][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:05:01,322][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:05:01,824][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:05:02,325][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:05:02,847][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:05:03,345][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:05:03,844][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:05:04,345][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:05:04,849][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:05:05,359][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:05:05,862][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:05:06,368][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:05:06,873][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:05:07,371][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:05:07,871][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:05:08,372][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:05:08,872][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:05:09,378][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:05:09,879][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:05:10,379][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:05:10,883][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10147 tokens.
[2025-11-13 04:05:11,666][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:33
[2025-11-13 04:05:12,434][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:05:12,436][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:05:12,438][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:05:13,487][__main__][INFO] - Iteration 397 took 54s (33.48% Gen, 64.60% Train). Generation: 18s, Training: 35s. Estimated remaining time: 39h 49m 0s. Estimated total time: 45h 44m 32s. Time estimates for 10 more iterations: 9m 8s, 100 more iterations: 1h 31m 29s, 500 more iterations: 7h 37m 25s.
[2025-11-13 04:05:13,489][__main__][INFO] - Starting iteration 397.
[2025-11-13 04:05:14,001][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 04:05:14,001][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:05:29,502][__main__][INFO] - Number of regex retries in iteration 397: 0
[2025-11-13 04:05:29,503][__main__][INFO] - agents played in iteration 397 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:05:30,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:05:30,353][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:05:30,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:05:30,404][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:05:30,405][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:05:30,406][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:05:31,134][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:05:31,587][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:05:32,092][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:05:32,589][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:05:33,086][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:05:33,590][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:05:34,087][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:05:34,587][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:05:35,088][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:05:35,589][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:05:36,094][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:05:36,599][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:05:37,100][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:05:37,623][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:05:38,126][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:05:38,650][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:05:39,156][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:05:39,655][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:05:40,157][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:05:40,655][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:05:41,155][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:05:41,654][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:05:42,154][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:05:42,662][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:05:43,162][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:05:43,660][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:05:44,160][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:05:44,660][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:05:45,166][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:05:45,665][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:05:46,162][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:05:46,663][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:05:47,164][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:05:47,669][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:05:48,175][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:05:48,680][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:05:49,188][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:05:49,691][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:05:50,193][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:05:50,697][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:05:51,201][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:05:51,701][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:05:52,203][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:05:52,704][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:05:53,204][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:05:53,705][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:05:54,205][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:05:54,703][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:05:55,201][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:05:55,704][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:05:56,205][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:05:56,707][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:05:57,213][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:05:57,718][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:05:58,231][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:05:58,732][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:05:59,234][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:05:59,737][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:06:00,237][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:06:00,751][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:06:01,251][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:06:01,751][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:06:02,252][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:06:02,753][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:06:03,253][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10135 tokens.
[2025-11-13 04:06:04,060][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.40%, ΔTime: 00:00:32
[2025-11-13 04:06:04,794][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:06:04,795][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:06:04,798][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:06:05,714][__main__][INFO] - Iteration 398 took 51s (29.98% Gen, 68.25% Train). Generation: 15s, Training: 35s. Estimated remaining time: 37h 9m 17s. Estimated total time: 43h 5m 41s. Time estimates for 10 more iterations: 8m 37s, 100 more iterations: 1h 26m 11s, 500 more iterations: 7h 10m 56s.
[2025-11-13 04:06:05,716][__main__][INFO] - Starting iteration 398.
[2025-11-13 04:06:06,193][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 04:06:06,193][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:06:17,226][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:06:21,974][__main__][INFO] - Number of regex retries in iteration 398: 1
[2025-11-13 04:06:21,974][__main__][INFO] - agents played in iteration 398 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:06:22,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:06:23,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:06:23,023][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:06:23,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:06:23,045][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:06:23,046][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:06:23,813][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:06:24,274][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:06:24,775][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:06:25,278][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:06:25,780][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:06:26,277][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:06:26,774][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:06:27,275][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:06:27,777][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:06:28,277][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:06:28,774][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:06:29,268][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:06:29,768][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:06:30,265][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:06:30,760][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:06:31,266][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:06:31,766][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:06:32,262][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:06:32,763][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:06:33,262][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:06:33,770][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:06:34,270][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:06:34,770][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:06:35,277][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:06:35,779][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:06:36,279][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:06:36,781][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:06:37,284][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:06:37,789][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:06:38,288][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:06:38,789][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:06:39,292][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:06:39,795][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:06:40,299][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:06:40,800][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:06:41,301][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:06:41,805][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:06:42,306][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:06:42,813][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:06:43,318][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:06:43,820][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:06:44,333][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:06:44,834][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:06:45,336][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:06:45,860][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:06:46,362][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:06:46,863][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:06:47,367][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:06:47,868][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:06:48,370][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:06:48,870][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:06:49,372][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:06:49,873][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:06:50,375][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:06:50,875][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:06:51,375][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:06:51,873][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:06:52,375][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:06:52,878][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:06:53,380][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:06:53,882][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:06:54,386][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:06:54,886][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:06:55,383][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:06:55,883][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10068 tokens.
[2025-11-13 04:06:56,704][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 04:06:57,462][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:06:57,464][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:06:57,466][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:06:58,384][__main__][INFO] - Iteration 399 took 52s (30.23% Gen, 68.00% Train). Generation: 15s, Training: 35s. Estimated remaining time: 37h 32m 19s. Estimated total time: 43h 29m 36s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 59s, 500 more iterations: 7h 14m 56s.
[2025-11-13 04:06:58,386][__main__][INFO] - Starting iteration 399.
[2025-11-13 04:06:58,890][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 04:06:58,892][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:07:19,387][__main__][INFO] - Number of regex retries in iteration 399: 0
[2025-11-13 04:07:19,387][__main__][INFO] - agents played in iteration 399 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:07:20,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:07:20,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:07:20,342][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:07:20,364][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:07:20,365][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:07:20,366][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:07:21,130][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:07:21,589][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:07:22,093][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:07:22,599][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:07:23,097][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:07:23,594][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:07:24,092][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:07:24,589][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:07:25,086][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:07:25,584][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:07:26,081][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:07:26,581][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:07:27,080][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:07:27,577][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:07:28,077][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:07:28,576][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:07:29,078][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:07:29,578][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:07:30,076][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:07:30,581][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:07:31,081][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:07:31,581][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:07:32,080][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:07:32,580][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:07:33,080][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:07:33,577][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:07:34,077][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:07:34,582][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:07:35,080][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:07:35,582][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:07:36,085][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:07:36,586][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:07:37,093][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:07:37,593][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:07:38,093][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:07:38,591][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:07:39,096][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:07:39,597][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:07:40,101][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:07:40,603][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:07:41,108][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:07:41,612][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:07:42,109][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:07:42,610][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:07:43,113][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:07:43,615][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:07:44,116][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:07:44,620][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:07:45,126][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:07:45,628][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:07:46,133][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:07:46,645][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:07:47,148][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:07:47,664][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:07:48,167][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:07:48,669][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:07:49,186][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:07:49,685][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:07:50,186][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:07:50,686][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:07:51,185][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:07:51,691][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:07:52,194][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:07:52,697][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:07:53,199][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10356 tokens.
[2025-11-13 04:07:53,973][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 04:07:54,749][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:07:54,751][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:07:54,752][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:07:55,739][__main__][INFO] - Iteration 400 took 56s (36.05% Gen, 62.21% Train). Generation: 20s, Training: 35s. Estimated remaining time: 41h 24m 15s. Estimated total time: 47h 22m 29s. Time estimates for 10 more iterations: 9m 28s, 100 more iterations: 1h 34m 44s, 500 more iterations: 7h 53m 44s.
[2025-11-13 04:07:55,741][__main__][INFO] - Starting iteration 400.
[2025-11-13 04:07:56,211][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 04:07:56,212][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:08:13,690][__main__][INFO] - Number of regex retries in iteration 400: 0
[2025-11-13 04:08:13,691][__main__][INFO] - agents played in iteration 400 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:08:14,491][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:08:14,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:08:14,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:08:14,568][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:08:14,569][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:08:14,570][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:08:15,302][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:08:15,758][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:08:16,262][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:08:16,781][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:08:17,288][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:08:17,785][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:08:18,290][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:08:18,786][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:08:19,292][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:08:19,792][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:08:20,290][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:08:20,793][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:08:21,294][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:08:21,794][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:08:22,289][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:08:22,786][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:08:23,295][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:08:23,792][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:08:24,286][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:08:24,781][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:08:25,277][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:08:25,781][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:08:26,277][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:08:26,772][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:08:27,276][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:08:27,773][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:08:28,268][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:08:28,766][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:08:29,262][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:08:29,774][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:08:30,270][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:08:30,767][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:08:31,281][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:08:31,782][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:08:32,281][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:08:32,779][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:08:33,278][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:08:33,793][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:08:34,293][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:08:34,798][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:08:35,299][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:08:35,801][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:08:36,304][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:08:36,802][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:08:37,303][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:08:37,806][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:08:38,313][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:08:38,813][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:08:39,320][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:08:39,821][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:08:40,324][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:08:40,824][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:08:41,324][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:08:41,826][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:08:42,328][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:08:42,828][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:08:43,331][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:08:43,831][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:08:44,333][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:08:44,834][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:08:45,333][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:08:45,833][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:08:46,338][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:08:46,841][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:08:47,340][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10137 tokens.
[2025-11-13 04:08:48,173][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 04:08:48,915][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:08:48,917][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:08:48,919][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:08:51,118][__main__][INFO] - Iteration 401 took 54s (31.83% Gen, 64.16% Train). Generation: 17s, Training: 35s. Estimated remaining time: 39h 46m 10s. Estimated total time: 45h 45m 20s. Time estimates for 10 more iterations: 9m 9s, 100 more iterations: 1h 31m 30s, 500 more iterations: 7h 37m 33s.
[2025-11-13 04:08:51,120][__main__][INFO] - Starting iteration 401.
[2025-11-13 04:08:51,596][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:08:51,597][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:09:10,126][__main__][INFO] - Number of regex retries in iteration 401: 0
[2025-11-13 04:09:10,127][__main__][INFO] - agents played in iteration 401 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:09:10,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:09:10,930][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:09:10,956][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:09:10,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:09:10,980][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:09:10,980][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:09:11,703][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:09:12,164][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:09:12,664][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:09:13,176][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:09:13,671][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:09:14,167][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:09:14,669][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:09:15,164][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:09:15,666][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:09:16,164][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:09:16,662][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:09:17,172][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:09:17,668][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:09:18,172][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:09:18,672][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:09:19,171][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:09:19,683][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:09:20,180][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:09:20,677][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:09:21,173][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:09:21,671][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:09:22,188][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:09:22,685][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:09:23,182][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:09:23,680][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:09:24,178][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:09:24,689][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:09:25,186][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:09:25,681][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:09:26,179][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:09:26,681][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:09:27,177][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:09:27,672][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:09:28,176][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:09:28,691][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:09:29,191][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:09:29,689][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:09:30,203][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:09:30,705][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:09:31,210][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:09:31,711][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:09:32,211][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:09:32,714][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:09:33,214][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:09:33,717][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:09:34,224][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:09:34,724][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:09:35,228][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:09:35,730][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:09:36,231][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:09:36,737][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:09:37,235][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:09:37,735][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:09:38,235][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:09:38,735][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:09:39,237][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:09:39,739][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:09:40,243][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:09:40,743][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:09:41,245][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:09:41,747][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:09:42,250][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:09:42,751][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:09:43,256][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:09:43,759][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10114 tokens.
[2025-11-13 04:09:44,603][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 04:09:45,358][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:09:45,359][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:09:45,361][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:09:46,672][__main__][INFO] - Iteration 402 took 55s (33.64% Gen, 63.97% Train). Generation: 18s, Training: 35s. Estimated remaining time: 39h 53m 45s. Estimated total time: 45h 53m 50s. Time estimates for 10 more iterations: 9m 10s, 100 more iterations: 1h 31m 47s, 500 more iterations: 7h 38m 58s.
[2025-11-13 04:09:46,674][__main__][INFO] - Starting iteration 402.
[2025-11-13 04:09:47,153][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:09:47,154][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:10:04,570][__main__][INFO] - Number of regex retries in iteration 402: 0
[2025-11-13 04:10:04,571][__main__][INFO] - agents played in iteration 402 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:10:05,365][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:10:05,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:10:05,418][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:10:05,442][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:10:05,442][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:10:05,443][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:10:06,161][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:10:06,616][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:10:07,119][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:10:07,616][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:10:08,113][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:10:08,610][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:10:09,107][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:10:09,615][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:10:10,112][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:10:10,610][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:10:11,108][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:10:11,604][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:10:12,110][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:10:12,610][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:10:13,110][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:10:13,620][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:10:14,118][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:10:14,624][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:10:15,125][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:10:15,623][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:10:16,124][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:10:16,624][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:10:17,122][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:10:17,624][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:10:18,120][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:10:18,619][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:10:19,113][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:10:19,610][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:10:20,113][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:10:20,610][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:10:21,105][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:10:21,606][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:10:22,102][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:10:22,599][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:10:23,101][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:10:23,601][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:10:24,114][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:10:24,615][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:10:25,116][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:10:25,620][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:10:26,121][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:10:26,626][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:10:27,126][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:10:27,629][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:10:28,132][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:10:28,632][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:10:29,131][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:10:29,655][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:10:30,158][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:10:30,657][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:10:31,158][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:10:31,660][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:10:32,161][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:10:32,660][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:10:33,159][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:10:33,657][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:10:34,155][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:10:34,658][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:10:35,160][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:10:35,662][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:10:36,170][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:10:36,673][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:10:37,179][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:10:37,679][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:10:38,179][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10122 tokens.
[2025-11-13 04:10:39,019][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 04:10:39,773][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:10:39,777][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:10:39,778][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:10:40,877][__main__][INFO] - Iteration 403 took 53s (32.42% Gen, 65.53% Train). Generation: 17s, Training: 35s. Estimated remaining time: 38h 45m 12s. Estimated total time: 44h 46m 11s. Time estimates for 10 more iterations: 8m 57s, 100 more iterations: 1h 29m 32s, 500 more iterations: 7h 27m 41s.
[2025-11-13 04:10:40,879][__main__][INFO] - Starting iteration 403.
[2025-11-13 04:10:41,357][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:10:41,357][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:10:55,427][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:10:57,903][__main__][INFO] - Number of regex retries in iteration 403: 1
[2025-11-13 04:10:57,904][__main__][INFO] - agents played in iteration 403 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:10:58,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:10:58,738][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:10:58,765][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:10:58,787][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:10:58,788][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:10:58,789][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:10:59,526][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:10:59,986][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:11:00,492][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:11:01,007][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:11:01,505][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:11:02,004][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:11:02,505][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:11:03,004][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:11:03,506][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:11:04,004][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:11:04,502][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:11:05,016][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:11:05,516][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:11:06,019][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:11:06,515][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:11:07,013][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:11:07,516][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:11:08,015][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:11:08,521][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:11:09,028][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:11:09,527][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:11:10,027][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:11:10,527][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:11:11,026][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:11:11,528][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:11:12,023][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:11:12,520][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:11:13,021][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:11:13,518][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:11:14,016][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:11:14,514][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:11:15,012][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:11:15,523][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:11:16,023][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:11:16,523][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:11:17,021][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:11:17,521][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:11:18,026][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:11:18,523][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:11:19,021][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:11:19,527][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:11:20,028][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:11:20,533][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:11:21,033][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:11:21,538][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:11:22,040][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:11:22,543][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:11:23,043][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:11:23,545][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:11:24,045][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:11:24,546][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:11:25,048][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:11:25,547][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:11:26,050][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:11:26,552][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:11:27,053][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:11:27,555][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:11:28,056][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:11:28,558][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:11:29,060][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:11:29,563][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:11:30,069][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:11:30,573][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:11:31,077][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:11:31,588][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10161 tokens.
[2025-11-13 04:11:32,408][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 04:11:33,154][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:11:33,155][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:11:33,157][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:11:34,542][__main__][INFO] - Iteration 404 took 53s (31.11% Gen, 66.28% Train). Generation: 16s, Training: 35s. Estimated remaining time: 38h 17m 23s. Estimated total time: 44h 19m 16s. Time estimates for 10 more iterations: 8m 51s, 100 more iterations: 1h 28m 38s, 500 more iterations: 7h 23m 12s.
[2025-11-13 04:11:34,547][__main__][INFO] - Starting iteration 404.
[2025-11-13 04:11:35,031][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:11:35,032][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:11:56,239][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:11:57,091][__main__][INFO] - Number of regex retries in iteration 404: 1
[2025-11-13 04:11:57,092][__main__][INFO] - agents played in iteration 404 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:11:57,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:11:57,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:11:58,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:11:58,025][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:11:58,025][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:11:58,027][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:11:58,755][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:11:59,210][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:11:59,721][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:12:00,223][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:12:00,720][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:12:01,240][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:12:01,736][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:12:02,240][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:12:02,738][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:12:03,236][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:12:03,740][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:12:04,241][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:12:04,744][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:12:05,245][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:12:05,745][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:12:06,247][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:12:06,744][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:12:07,262][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:12:07,770][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:12:08,267][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:12:08,768][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:12:09,265][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:12:09,761][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:12:10,261][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:12:10,759][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:12:11,253][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:12:11,751][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:12:12,253][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:12:12,758][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:12:13,259][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:12:13,761][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:12:14,265][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:12:14,763][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:12:15,265][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:12:15,767][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:12:16,270][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:12:16,773][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:12:17,274][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:12:17,777][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:12:18,282][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:12:18,782][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:12:19,290][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:12:19,791][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:12:20,291][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:12:20,795][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:12:21,297][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:12:21,804][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:12:22,318][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:12:22,820][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:12:23,334][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:12:23,836][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:12:24,340][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:12:24,847][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:12:25,349][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:12:25,856][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:12:26,359][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:12:26,862][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:12:27,364][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:12:27,864][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:12:28,363][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:12:28,864][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:12:29,365][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:12:29,868][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:12:30,369][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:12:30,871][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10114 tokens.
[2025-11-13 04:12:31,694][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 04:12:32,455][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:12:32,458][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:12:32,460][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:12:33,602][__main__][INFO] - Iteration 405 took 58s (37.66% Gen, 60.39% Train). Generation: 22s, Training: 35s. Estimated remaining time: 42h 45m 43s. Estimated total time: 48h 48m 35s. Time estimates for 10 more iterations: 9m 45s, 100 more iterations: 1h 37m 37s, 500 more iterations: 8h 8m 5s.
[2025-11-13 04:12:33,604][__main__][INFO] - Starting iteration 405.
[2025-11-13 04:12:34,082][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:12:34,083][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:12:53,490][__main__][INFO] - Number of regex retries in iteration 405: 0
[2025-11-13 04:12:53,491][__main__][INFO] - agents played in iteration 405 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:12:54,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:12:54,351][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:12:54,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:12:54,398][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:12:54,398][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:12:54,399][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:12:55,115][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:12:55,709][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:12:56,182][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:12:56,681][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:12:57,179][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:12:57,683][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:12:58,180][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:12:58,677][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:12:59,181][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:12:59,677][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:13:00,173][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:13:00,672][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:13:01,170][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:13:01,670][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:13:02,171][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:13:02,672][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:13:03,171][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:13:03,666][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:13:04,163][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:13:04,660][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:13:05,158][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:13:05,660][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:13:06,160][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:13:06,660][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:13:07,163][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:13:07,664][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:13:08,163][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:13:08,661][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:13:09,161][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:13:09,659][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:13:10,157][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:13:10,656][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:13:11,156][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:13:11,652][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:13:12,161][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:13:12,662][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:13:13,161][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:13:13,661][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:13:14,162][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:13:14,665][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:13:15,165][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:13:15,661][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:13:16,162][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:13:16,659][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:13:17,158][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:13:17,661][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:13:18,161][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:13:18,662][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:13:19,162][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:13:19,663][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:13:20,167][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:13:20,665][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:13:21,170][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:13:21,672][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:13:22,171][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:13:22,673][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:13:23,172][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:13:23,676][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:13:24,178][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:13:24,680][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:13:25,186][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:13:25,690][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:13:26,192][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:13:26,695][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:13:27,195][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10095 tokens.
[2025-11-13 04:13:27,985][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 04:13:28,726][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:13:28,728][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:13:28,730][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:13:29,766][__main__][INFO] - Iteration 406 took 55s (34.85% Gen, 63.28% Train). Generation: 19s, Training: 35s. Estimated remaining time: 40h 20m 25s. Estimated total time: 46h 24m 13s. Time estimates for 10 more iterations: 9m 16s, 100 more iterations: 1h 32m 48s, 500 more iterations: 7h 44m 2s.
[2025-11-13 04:13:29,768][__main__][INFO] - Starting iteration 406.
[2025-11-13 04:13:30,237][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:13:30,238][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:13:44,195][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:13:45,063][__main__][INFO] - Number of regex retries in iteration 406: 1
[2025-11-13 04:13:45,064][__main__][INFO] - agents played in iteration 406 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:13:45,957][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:13:45,983][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:13:46,008][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:13:46,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:13:46,031][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:13:46,032][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:13:46,757][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:13:47,213][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:13:47,719][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:13:48,219][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:13:48,720][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:13:49,228][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:13:49,729][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:13:50,237][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:13:50,733][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:13:51,234][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:13:51,743][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:13:52,241][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:13:52,744][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:13:53,243][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:13:53,742][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:13:54,259][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:13:54,757][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:13:55,258][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:13:55,756][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:13:56,258][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:13:56,770][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:13:57,266][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:13:57,765][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:13:58,268][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:13:58,768][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:13:59,274][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:13:59,771][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:14:00,278][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:14:00,777][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:14:01,274][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:14:01,776][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:14:02,273][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:14:02,772][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:14:03,282][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:14:03,778][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:14:04,276][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:14:04,780][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:14:05,275][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:14:05,774][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:14:06,271][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:14:06,767][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:14:07,275][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:14:07,771][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:14:08,268][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:14:08,765][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:14:09,263][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:14:09,773][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:14:10,274][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:14:10,779][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:14:11,282][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:14:11,787][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:14:12,288][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:14:12,788][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:14:13,289][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:14:13,791][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:14:14,292][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:14:14,796][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:14:15,305][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:14:15,806][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:14:16,308][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:14:16,809][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:14:17,312][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:14:17,812][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:14:18,311][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:14:18,815][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10114 tokens.
[2025-11-13 04:14:19,636][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 04:14:20,395][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:14:20,396][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:14:20,399][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:14:22,341][__main__][INFO] - Iteration 407 took 52s (28.45% Gen, 67.82% Train). Generation: 14s, Training: 35s. Estimated remaining time: 37h 20m 32s. Estimated total time: 43h 25m 13s. Time estimates for 10 more iterations: 8m 41s, 100 more iterations: 1h 26m 50s, 500 more iterations: 7h 14m 12s.
[2025-11-13 04:14:22,346][__main__][INFO] - Starting iteration 407.
[2025-11-13 04:14:22,836][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:14:22,836][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:14:43,617][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:14:56,667][__main__][INFO] - Number of regex retries in iteration 407: 1
[2025-11-13 04:14:56,667][__main__][INFO] - agents played in iteration 407 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:14:57,513][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:14:57,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:14:57,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:14:57,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:14:57,586][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:14:57,586][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:14:58,301][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:14:58,754][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:14:59,262][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:14:59,774][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:15:00,276][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:15:00,772][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:15:01,272][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:15:01,774][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:15:02,284][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:15:02,783][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:15:03,282][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:15:03,781][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:15:04,279][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:15:04,777][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:15:05,281][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:15:05,782][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:15:06,295][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:15:06,798][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:15:07,300][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:15:07,806][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:15:08,310][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:15:08,816][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:15:09,318][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:15:09,822][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:15:10,326][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:15:10,829][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:15:11,332][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:15:11,834][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:15:12,338][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:15:12,844][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:15:13,345][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:15:13,847][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:15:14,357][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:15:14,862][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:15:15,369][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:15:15,869][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:15:16,374][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:15:16,894][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:15:17,397][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:15:17,901][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:15:18,403][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:15:18,903][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:15:19,407][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:15:19,913][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:15:20,418][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:15:20,921][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:15:21,424][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:15:21,929][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:15:22,429][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:15:22,935][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:15:23,437][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:15:23,938][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:15:24,440][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:15:24,942][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:15:25,445][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:15:25,947][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:15:26,449][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:15:26,947][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:15:27,458][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:15:27,958][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:15:28,457][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:15:28,955][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:15:29,458][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:15:29,967][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:15:30,465][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10064 tokens.
[2025-11-13 04:15:31,300][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:33
[2025-11-13 04:15:32,050][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:15:32,052][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:15:32,053][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:15:33,150][__main__][INFO] - Iteration 408 took 1m 10s (48.11% Gen, 50.33% Train). Generation: 33s, Training: 35s. Estimated remaining time: 52h 29m 52s. Estimated total time: 58h 35m 44s. Time estimates for 10 more iterations: 11m 43s, 100 more iterations: 1h 57m 11s, 500 more iterations: 9h 45m 57s.
[2025-11-13 04:15:33,152][__main__][INFO] - Starting iteration 408.
[2025-11-13 04:15:33,662][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:15:33,662][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:15:50,439][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:15:58,457][__main__][INFO] - Number of regex retries in iteration 408: 1
[2025-11-13 04:15:58,458][__main__][INFO] - agents played in iteration 408 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:15:59,310][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:15:59,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:15:59,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:15:59,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:15:59,387][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:15:59,388][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:16:00,101][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:16:00,569][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:16:01,071][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:16:01,575][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:16:02,078][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:16:02,577][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:16:03,091][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:16:03,588][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:16:04,085][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:16:04,581][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:16:05,080][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:16:05,579][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:16:06,080][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:16:06,578][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:16:07,080][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:16:07,580][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:16:08,094][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:16:08,593][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:16:09,097][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:16:09,603][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:16:10,104][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:16:10,607][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:16:11,108][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:16:11,611][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:16:12,118][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:16:12,622][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:16:13,125][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:16:13,628][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:16:14,131][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:16:14,641][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:16:15,147][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:16:15,648][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:16:16,154][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:16:16,660][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:16:17,160][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:16:17,662][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:16:18,166][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:16:18,672][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:16:19,171][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:16:19,676][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:16:20,176][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:16:20,680][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:16:21,193][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:16:21,699][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:16:22,202][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:16:22,705][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:16:23,209][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:16:24,402][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:16:25,612][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:16:26,127][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:16:26,657][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:16:27,162][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:16:27,659][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:16:28,158][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:16:28,660][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:16:29,162][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:16:29,661][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:16:30,160][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:16:30,660][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:16:31,159][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:16:31,657][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:16:32,154][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:16:32,655][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:16:33,156][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:16:33,655][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10154 tokens.
[2025-11-13 04:16:34,543][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:34
[2025-11-13 04:16:35,211][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:16:35,213][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:16:35,214][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:16:36,186][__main__][INFO] - Iteration 409 took 1m 2s (39.66% Gen, 58.79% Train). Generation: 24s, Training: 36s. Estimated remaining time: 45h 59m 18s. Estimated total time: 52h 6m 13s. Time estimates for 10 more iterations: 10m 25s, 100 more iterations: 1h 44m 12s, 500 more iterations: 8h 41m 2s.
[2025-11-13 04:16:36,188][__main__][INFO] - Starting iteration 409.
[2025-11-13 04:16:36,693][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:16:36,694][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:16:57,461][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:16:57,546][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:16:58,141][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:17:08,704][__main__][INFO] - Number of regex retries in iteration 409: 3
[2025-11-13 04:17:08,705][__main__][INFO] - agents played in iteration 409 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:17:09,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:17:09,589][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:17:09,614][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:17:09,637][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:17:09,639][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:17:09,639][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:17:10,487][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:17:10,948][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:17:11,459][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:17:11,963][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:17:12,469][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:17:12,978][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:17:13,483][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:17:13,999][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:17:14,501][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:17:15,007][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:17:15,511][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:17:16,014][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:17:16,529][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:17:17,034][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:17:17,539][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:17:18,047][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:17:18,555][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:17:19,061][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:17:19,562][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:17:20,068][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:17:20,570][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:17:21,072][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:17:21,572][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:17:22,080][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:17:22,582][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:17:23,108][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:17:23,610][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:17:24,110][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:17:24,626][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:17:25,132][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:17:25,635][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:17:26,141][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:17:26,645][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:17:27,149][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:17:27,648][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:17:28,152][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:17:28,655][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:17:29,157][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:17:29,660][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:17:30,161][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:17:30,666][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:17:31,165][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:17:31,669][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:17:32,172][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:17:32,672][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:17:33,171][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:17:33,669][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:17:34,167][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:17:34,667][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:17:35,174][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:17:35,673][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:17:36,174][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:17:36,671][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:17:37,167][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:17:37,665][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:17:38,163][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:17:38,660][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:17:39,163][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:17:39,662][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:17:40,162][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:17:40,659][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:17:41,161][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:17:41,664][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:17:42,167][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:17:42,665][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10241 tokens.
[2025-11-13 04:17:43,453][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 04:17:44,240][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:17:44,242][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:17:44,245][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:17:45,302][__main__][INFO] - Iteration 410 took 1m 8s (46.66% Gen, 51.80% Train). Generation: 32s, Training: 35s. Estimated remaining time: 51h 2m 25s. Estimated total time: 57h 10m 28s. Time estimates for 10 more iterations: 11m 26s, 100 more iterations: 1h 54m 20s, 500 more iterations: 9h 31m 44s.
[2025-11-13 04:17:45,304][__main__][INFO] - Starting iteration 410.
[2025-11-13 04:17:45,787][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 04:17:45,788][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:18:13,325][__main__][INFO] - Number of regex retries in iteration 410: 0
[2025-11-13 04:18:13,325][__main__][INFO] - agents played in iteration 410 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:18:14,188][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:18:14,211][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:18:14,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:18:14,273][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:18:14,274][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:18:14,275][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:18:15,097][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:18:15,555][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:18:16,065][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:18:16,568][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:18:17,076][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:18:17,580][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:18:18,082][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:18:18,583][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:18:19,086][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:18:19,587][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:18:20,100][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:18:20,605][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:18:21,114][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:18:21,617][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:18:22,117][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:18:22,624][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:18:23,125][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:18:23,633][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:18:24,139][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:18:24,639][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:18:25,144][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:18:25,646][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:18:26,147][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:18:26,651][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:18:27,155][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:18:27,655][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:18:28,158][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:18:28,658][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:18:29,168][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:18:29,667][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:18:30,166][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:18:30,669][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:18:31,171][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:18:31,675][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:18:32,179][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:18:32,681][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:18:33,209][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:18:33,717][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:18:34,222][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:18:34,728][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:18:35,231][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:18:35,742][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:18:36,245][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:18:36,747][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:18:37,248][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:18:37,746][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:18:38,247][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:18:38,750][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:18:39,253][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:18:39,756][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:18:40,254][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:18:40,754][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:18:41,255][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:18:41,756][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:18:42,261][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:18:42,763][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:18:43,267][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:18:43,772][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:18:44,270][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:18:44,767][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:18:45,264][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:18:45,762][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:18:46,263][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:18:46,766][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:18:47,267][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10143 tokens.
[2025-11-13 04:18:48,013][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 04:18:48,799][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:18:48,801][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:18:48,802][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:18:52,599][__main__][INFO] - Iteration 411 took 1m 6s (41.22% Gen, 53.10% Train). Generation: 27s, Training: 35s. Estimated remaining time: 49h 31m 24s. Estimated total time: 55h 40m 35s. Time estimates for 10 more iterations: 11m 8s, 100 more iterations: 1h 51m 21s, 500 more iterations: 9h 16m 45s.
[2025-11-13 04:18:52,604][__main__][INFO] - Starting iteration 411.
[2025-11-13 04:18:53,094][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:18:53,094][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:19:28,820][__main__][INFO] - Number of regex retries in iteration 411: 0
[2025-11-13 04:19:28,820][__main__][INFO] - agents played in iteration 411 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:19:29,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:19:29,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:19:29,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:19:29,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:19:29,779][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:19:29,780][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:19:30,643][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:19:31,106][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:19:31,611][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:19:32,128][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:19:32,630][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:19:33,141][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:19:33,642][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:19:34,146][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:19:34,654][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:19:35,159][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:19:35,665][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:19:36,169][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:19:36,672][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:19:37,184][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:19:37,686][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:19:38,190][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:19:38,692][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:19:39,193][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:19:39,693][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:19:40,195][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:19:40,695][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:19:41,197][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:19:41,696][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:19:42,197][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:19:42,699][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:19:43,203][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:19:43,716][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:19:44,214][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:19:44,713][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:19:45,223][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:19:45,726][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:19:46,242][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:19:46,741][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:19:47,244][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:19:47,750][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:19:48,250][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:19:48,780][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:19:49,288][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:19:49,791][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:19:50,292][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:19:50,793][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:19:51,295][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:19:51,793][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:19:52,293][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:19:52,793][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:19:53,290][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:19:53,791][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:19:54,294][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:19:54,791][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:19:55,287][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:19:55,784][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:19:56,282][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:19:56,780][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:19:57,277][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:19:57,773][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:19:58,272][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:19:58,767][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:19:59,265][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:19:59,764][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:20:00,262][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:20:00,761][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:20:01,261][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:20:01,758][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:20:02,260][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:20:02,757][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10087 tokens.
[2025-11-13 04:20:03,501][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 04:20:04,134][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:20:04,135][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:20:04,137][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:20:05,192][__main__][INFO] - Iteration 412 took 1m 12s (49.55% Gen, 48.98% Train). Generation: 35s, Training: 35s. Estimated remaining time: 53h 54m 33s. Estimated total time: 60h 4m 57s. Time estimates for 10 more iterations: 12m 0s, 100 more iterations: 2h 0m 9s, 500 more iterations: 10h 0m 49s.
[2025-11-13 04:20:05,194][__main__][INFO] - Starting iteration 412.
[2025-11-13 04:20:05,660][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:20:05,661][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:20:16,735][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:20:17,749][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:20:28,359][__main__][INFO] - Number of regex retries in iteration 412: 2
[2025-11-13 04:20:28,360][__main__][INFO] - agents played in iteration 412 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:20:29,205][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:20:29,229][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:20:29,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:20:29,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:20:29,276][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:20:29,277][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:20:30,123][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:20:30,590][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:20:31,096][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:20:31,607][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:20:32,113][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:20:32,623][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:20:33,127][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:20:33,629][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:20:34,132][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:20:34,636][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:20:35,141][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:20:35,651][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:20:36,155][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:20:36,672][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:20:37,221][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:20:38,507][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:20:39,002][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:20:39,508][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:20:40,009][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:20:40,512][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:20:41,020][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:20:41,517][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:20:42,023][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:20:42,523][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:20:43,025][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:20:43,529][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:20:44,030][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:20:44,541][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:20:45,041][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:20:45,538][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:20:46,043][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:20:46,544][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:20:47,056][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:20:47,556][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:20:48,057][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:20:48,570][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:20:49,069][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:20:49,571][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:20:50,070][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:20:50,569][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:20:51,074][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:20:51,575][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:20:52,073][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:20:52,578][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:20:53,080][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:20:53,581][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:20:54,082][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:20:54,582][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:20:55,092][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:20:55,591][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:20:56,092][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:20:56,592][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:20:57,088][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:20:57,583][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:20:58,079][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:20:58,576][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:20:59,078][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:20:59,579][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:21:00,078][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:21:00,578][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:21:01,073][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:21:01,569][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:21:02,064][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:21:02,561][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:21:03,076][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10111 tokens.
[2025-11-13 04:21:03,843][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:33
[2025-11-13 04:21:04,531][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:21:04,533][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:21:04,534][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:21:05,498][__main__][INFO] - Iteration 413 took 59s (37.93% Gen, 60.45% Train). Generation: 22s, Training: 36s. Estimated remaining time: 43h 40m 29s. Estimated total time: 49h 51m 53s. Time estimates for 10 more iterations: 9m 58s, 100 more iterations: 1h 39m 43s, 500 more iterations: 8h 18m 38s.
[2025-11-13 04:21:05,500][__main__][INFO] - Starting iteration 413.
[2025-11-13 04:21:05,985][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:21:05,985][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:21:39,727][__main__][INFO] - Number of regex retries in iteration 413: 0
[2025-11-13 04:21:39,728][__main__][INFO] - agents played in iteration 413 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:21:40,587][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:21:40,614][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:21:40,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:21:40,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:21:40,664][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:21:40,665][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:21:41,518][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:21:41,979][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:21:42,489][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:21:43,001][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:21:43,504][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:21:44,009][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:21:44,511][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:21:45,014][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:21:45,517][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:21:46,018][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:21:46,525][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:21:47,025][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:21:47,525][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:21:48,026][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:21:48,525][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:21:49,026][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:21:49,538][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:21:50,040][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:21:50,546][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:21:51,046][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:21:51,547][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:21:52,053][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:21:52,556][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:21:53,060][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:21:53,562][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:21:54,066][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:21:54,584][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:21:55,088][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:21:55,594][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:21:56,096][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:21:56,595][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:21:57,091][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:21:57,588][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:21:58,086][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:21:58,589][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:21:59,090][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:21:59,587][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:22:00,084][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:22:00,582][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:22:01,087][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:22:01,586][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:22:02,086][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:22:02,587][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:22:03,082][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:22:03,582][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:22:04,080][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:22:04,577][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:22:05,082][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:22:05,579][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:22:06,079][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:22:06,580][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:22:07,078][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:22:07,583][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:22:08,104][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:22:08,602][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:22:09,102][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:22:09,601][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:22:10,100][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:22:10,598][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:22:11,102][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:22:11,600][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:22:12,099][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:22:12,599][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:22:13,097][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:22:13,597][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10313 tokens.
[2025-11-13 04:22:14,330][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.35%, ΔTime: 00:00:32
[2025-11-13 04:22:15,106][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:22:15,109][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:22:15,110][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:22:16,139][__main__][INFO] - Iteration 414 took 1m 10s (48.10% Gen, 50.43% Train). Generation: 33s, Training: 35s. Estimated remaining time: 52h 15m 11s. Estimated total time: 58h 27m 45s. Time estimates for 10 more iterations: 11m 41s, 100 more iterations: 1h 56m 55s, 500 more iterations: 9h 44m 37s.
[2025-11-13 04:22:16,142][__main__][INFO] - Starting iteration 414.
[2025-11-13 04:22:16,624][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:22:16,625][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:22:25,030][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:22:35,729][__main__][INFO] - Number of regex retries in iteration 414: 1
[2025-11-13 04:22:35,730][__main__][INFO] - agents played in iteration 414 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:22:36,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:22:36,580][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:22:36,606][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:22:36,629][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:22:36,630][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:22:36,632][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:22:37,468][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:22:37,934][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:22:38,442][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:22:38,945][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:22:39,465][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:22:39,968][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:22:40,481][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:22:40,983][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:22:41,484][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:22:41,992][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:22:42,493][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:22:42,997][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:22:43,499][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:22:43,997][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:22:44,501][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:22:45,004][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:22:45,504][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:22:46,004][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:22:46,505][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:22:47,006][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:22:47,507][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:22:48,012][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:22:48,514][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:22:49,016][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:22:49,518][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:22:50,017][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:22:50,517][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:22:51,022][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:22:51,528][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:22:52,027][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:22:52,530][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:22:53,028][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:22:53,537][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:22:54,040][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:22:54,546][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:22:55,064][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:22:55,575][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:22:56,075][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:22:56,573][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:22:57,074][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:22:57,577][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:22:58,080][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:22:58,577][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:22:59,079][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:22:59,582][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:23:00,084][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:23:00,582][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:23:01,082][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:23:01,592][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:23:02,094][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:23:02,598][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:23:03,098][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:23:03,598][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:23:04,095][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:23:04,592][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:23:05,089][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:23:05,586][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:23:06,082][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:23:06,578][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:23:07,074][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:23:07,572][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:23:08,074][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:23:08,569][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:23:09,064][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:23:09,565][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10050 tokens.
[2025-11-13 04:23:10,314][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.99%, Current % of VRAM taken: 60.25%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 04:23:11,106][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:23:11,108][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:23:11,110][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:23:12,161][__main__][INFO] - Iteration 415 took 55s (34.40% Gen, 63.71% Train). Generation: 19s, Training: 35s. Estimated remaining time: 40h 3m 19s. Estimated total time: 46h 16m 50s. Time estimates for 10 more iterations: 9m 15s, 100 more iterations: 1h 32m 33s, 500 more iterations: 7h 42m 48s.
[2025-11-13 04:23:12,163][__main__][INFO] - Starting iteration 415.
[2025-11-13 04:23:12,632][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:23:12,632][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:23:35,985][__main__][INFO] - Number of regex retries in iteration 415: 0
[2025-11-13 04:23:35,987][__main__][INFO] - agents played in iteration 415 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:23:37,139][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:23:37,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:23:37,220][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:23:37,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:23:37,248][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:23:37,250][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:23:38,096][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:23:38,562][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:23:39,069][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:23:39,593][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:23:40,100][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:23:40,604][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:23:41,107][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:23:41,609][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:23:42,112][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:23:42,616][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:23:43,123][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:23:43,628][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:23:44,129][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:23:44,632][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:23:45,138][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:23:45,641][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:23:46,149][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:23:46,651][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:23:47,156][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:23:47,657][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:23:48,158][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:23:48,670][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:23:49,172][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:23:49,677][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:23:50,182][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:23:50,686][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:23:51,188][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:23:51,690][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:23:52,190][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:23:52,701][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:23:53,202][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:23:53,705][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:23:54,215][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:23:54,723][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:23:55,229][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:23:55,731][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:23:56,233][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:23:56,734][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:23:57,235][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:23:57,739][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:23:58,248][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:23:58,747][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:23:59,256][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:23:59,754][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:24:00,254][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:24:00,769][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:24:01,272][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:24:01,785][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:24:02,287][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:24:02,790][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:24:03,297][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:24:03,797][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:24:04,295][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:24:04,795][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:24:05,291][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:24:05,796][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:24:06,292][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:24:06,789][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:24:07,290][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:24:07,788][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:24:08,289][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:24:08,786][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:24:09,281][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:24:09,794][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:24:10,290][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10158 tokens.
[2025-11-13 04:24:11,042][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:32
[2025-11-13 04:24:11,675][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:24:11,677][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:24:11,679][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:24:12,618][__main__][INFO] - Iteration 416 took 59s (38.93% Gen, 59.50% Train). Generation: 23s, Training: 35s. Estimated remaining time: 43h 44m 48s. Estimated total time: 49h 59m 19s. Time estimates for 10 more iterations: 9m 59s, 100 more iterations: 1h 39m 58s, 500 more iterations: 8h 19m 53s.
[2025-11-13 04:24:12,620][__main__][INFO] - Starting iteration 416.
[2025-11-13 04:24:13,105][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:24:13,106][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:24:36,407][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:24:37,485][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:24:46,824][__main__][INFO] - Number of regex retries in iteration 416: 2
[2025-11-13 04:24:46,825][__main__][INFO] - agents played in iteration 416 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:24:47,647][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:24:47,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:24:47,693][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:24:47,731][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:24:47,731][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:24:47,732][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:24:48,561][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:24:49,027][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:24:49,539][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:24:50,039][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:24:50,541][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:24:51,044][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:24:51,546][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:24:52,048][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:24:52,550][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:24:53,057][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:24:53,557][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:24:54,058][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:24:54,565][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:24:55,068][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:24:55,570][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:24:56,077][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:24:56,575][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:24:57,082][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:24:57,580][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:24:58,084][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:24:58,599][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:24:59,105][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:24:59,612][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:25:00,113][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:25:00,612][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:25:01,114][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:25:01,618][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:25:02,121][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:25:02,627][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:25:03,126][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:25:03,627][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:25:04,126][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:25:04,625][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:25:05,129][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:25:05,630][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:25:06,131][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:25:06,632][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:25:07,131][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:25:07,632][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:25:08,135][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:25:08,638][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:25:09,138][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:25:09,639][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:25:10,134][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:25:10,631][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:25:11,127][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:25:11,626][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:25:12,122][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:25:12,619][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:25:13,120][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:25:13,620][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:25:14,119][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:25:14,620][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:25:15,117][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:25:15,621][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:25:16,120][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:25:16,622][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:25:17,125][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:25:17,624][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:25:18,122][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:25:18,620][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:25:19,120][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:25:19,619][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:25:20,120][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:25:20,619][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10203 tokens.
[2025-11-13 04:25:21,358][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 04:25:22,097][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:25:22,099][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:25:22,100][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:25:23,176][__main__][INFO] - Iteration 417 took 1m 10s (48.12% Gen, 50.34% Train). Generation: 33s, Training: 35s. Estimated remaining time: 52h 7m 51s. Estimated total time: 58h 23m 33s. Time estimates for 10 more iterations: 11m 40s, 100 more iterations: 1h 56m 47s, 500 more iterations: 9h 43m 55s.
[2025-11-13 04:25:23,178][__main__][INFO] - Starting iteration 417.
[2025-11-13 04:25:23,649][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:25:23,649][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:25:34,630][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:25:47,430][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:25:48,537][__main__][INFO] - Number of regex retries in iteration 417: 2
[2025-11-13 04:25:48,537][__main__][INFO] - agents played in iteration 417 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:25:49,405][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:25:49,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:25:49,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:25:49,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:25:49,487][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:25:49,488][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:25:50,339][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:25:50,806][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:25:51,316][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:25:51,827][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:25:52,332][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:25:52,833][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:25:53,338][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:25:53,844][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:25:54,349][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:25:54,852][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:25:55,353][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:25:55,854][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:25:56,356][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:25:56,862][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:25:57,363][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:25:57,864][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:25:58,373][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:25:58,876][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:25:59,404][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:25:59,908][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:26:00,415][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:26:00,917][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:26:01,423][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:26:01,929][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:26:02,432][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:26:02,933][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:26:03,437][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:26:03,939][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:26:04,442][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:26:04,946][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:26:05,449][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:26:05,956][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:26:06,454][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:26:06,954][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:26:07,454][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:26:07,953][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:26:08,453][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:26:08,953][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:26:09,453][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:26:09,950][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:26:10,448][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:26:10,946][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:26:11,448][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:26:11,944][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:26:12,445][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:26:12,944][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:26:13,445][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:26:13,946][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:26:14,447][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:26:14,947][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:26:15,447][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:26:15,947][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:26:16,445][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:26:16,948][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:26:17,447][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:26:17,946][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:26:18,442][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:26:18,941][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:26:19,440][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:26:19,940][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:26:20,437][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:26:20,934][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:26:21,433][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:26:21,930][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:26:22,429][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10171 tokens.
[2025-11-13 04:26:23,167][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:32
[2025-11-13 04:26:23,823][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:26:23,825][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:26:23,827][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:26:24,836][__main__][INFO] - Iteration 418 took 1m 1s (40.67% Gen, 57.67% Train). Generation: 24s, Training: 35s. Estimated remaining time: 44h 42m 40s. Estimated total time: 50h 59m 23s. Time estimates for 10 more iterations: 10m 11s, 100 more iterations: 1h 41m 58s, 500 more iterations: 8h 29m 53s.
[2025-11-13 04:26:24,838][__main__][INFO] - Starting iteration 418.
[2025-11-13 04:26:25,342][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:26:25,343][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:26:44,956][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:26:57,218][__main__][INFO] - Number of regex retries in iteration 418: 1
[2025-11-13 04:26:57,219][__main__][INFO] - agents played in iteration 418 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:26:58,090][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:26:58,115][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:26:58,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:26:58,175][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:26:58,176][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:26:58,176][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:26:58,986][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:26:59,442][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:26:59,948][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:27:00,448][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:27:00,953][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:27:01,454][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:27:01,957][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:27:02,461][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:27:02,963][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:27:03,467][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:27:03,968][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:27:04,472][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:27:04,983][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:27:05,485][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:27:05,989][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:27:06,494][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:27:06,992][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:27:07,517][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:27:08,020][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:27:08,521][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:27:09,029][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:27:09,535][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:27:10,066][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:27:10,571][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:27:11,077][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:27:11,578][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:27:12,075][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:27:12,573][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:27:13,071][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:27:13,569][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:27:14,067][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:27:14,567][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:27:15,066][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:27:15,567][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:27:16,064][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:27:16,568][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:27:17,066][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:27:17,563][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:27:18,064][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:27:18,564][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:27:19,064][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:27:19,564][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:27:20,066][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:27:20,562][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:27:21,058][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:27:21,559][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:27:22,055][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:27:22,551][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:27:23,048][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:27:23,545][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:27:24,045][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:27:24,541][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:27:25,037][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:27:25,533][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:27:26,035][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:27:26,532][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:27:27,030][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:27:27,530][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:27:28,030][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:27:28,528][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:27:29,025][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:27:29,525][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:27:30,029][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:27:30,529][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:27:31,030][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10159 tokens.
[2025-11-13 04:27:31,759][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 04:27:32,503][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:27:32,505][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:27:32,506][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:27:33,507][__main__][INFO] - Iteration 419 took 1m 8s (46.76% Gen, 51.77% Train). Generation: 31s, Training: 35s. Estimated remaining time: 50h 30m 24s. Estimated total time: 56h 48m 16s. Time estimates for 10 more iterations: 11m 21s, 100 more iterations: 1h 53m 36s, 500 more iterations: 9h 28m 2s.
[2025-11-13 04:27:33,509][__main__][INFO] - Starting iteration 419.
[2025-11-13 04:27:33,980][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:27:33,980][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:28:00,760][__main__][INFO] - Number of regex retries in iteration 419: 0
[2025-11-13 04:28:00,762][__main__][INFO] - agents played in iteration 419 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:28:01,686][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:28:01,712][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:28:01,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:28:01,760][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:28:01,761][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:28:01,762][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:28:02,605][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:28:03,062][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:28:03,571][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:28:04,073][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:28:04,578][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:28:05,077][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:28:05,582][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:28:06,084][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:28:06,600][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:28:07,102][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:28:07,609][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:28:08,113][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:28:08,620][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:28:09,125][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:28:09,626][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:28:10,126][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:28:10,628][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:28:11,131][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:28:11,633][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:28:12,135][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:28:12,638][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:28:13,144][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:28:13,644][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:28:14,144][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:28:14,656][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:28:15,156][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:28:15,659][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:28:16,166][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:28:16,666][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:28:17,166][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:28:17,665][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:28:18,165][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:28:18,665][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:28:19,167][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:28:19,665][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:28:20,162][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:28:20,660][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:28:21,158][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:28:21,655][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:28:22,152][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:28:22,650][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:28:23,147][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:28:23,646][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:28:24,143][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:28:24,641][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:28:25,142][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:28:25,640][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:28:26,141][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:28:26,646][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:28:27,147][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:28:27,648][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:28:28,146][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:28:28,645][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:28:29,144][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:28:29,645][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:28:30,145][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:28:30,644][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:28:31,141][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:28:31,641][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:28:32,138][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:28:32,634][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:28:33,137][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:28:33,631][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:28:34,127][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:28:34,622][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10115 tokens.
[2025-11-13 04:28:35,376][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.00%, Current % of VRAM taken: 60.26%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 04:28:36,014][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:28:36,015][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:28:36,017][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:28:36,963][__main__][INFO] - Iteration 420 took 1m 2s (42.52% Gen, 55.97% Train). Generation: 26s, Training: 35s. Estimated remaining time: 46h 10m 15s. Estimated total time: 52h 29m 10s. Time estimates for 10 more iterations: 10m 29s, 100 more iterations: 1h 44m 58s, 500 more iterations: 8h 44m 51s.
[2025-11-13 04:28:36,965][__main__][INFO] - Starting iteration 420.
[2025-11-13 04:28:37,457][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 04:28:37,458][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:28:55,347][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:29:05,630][__main__][INFO] - Number of regex retries in iteration 420: 1
[2025-11-13 04:29:05,631][__main__][INFO] - agents played in iteration 420 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:29:06,445][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:29:06,471][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:29:06,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:29:06,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:29:06,519][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:29:06,519][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:29:07,337][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:29:07,795][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:29:08,318][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:29:08,824][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:29:09,327][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:29:09,831][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:29:10,334][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:29:10,845][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:29:11,349][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:29:11,848][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:29:12,352][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:29:12,852][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:29:13,361][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:29:13,862][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:29:14,367][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:29:14,871][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:29:15,372][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:29:15,872][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:29:16,378][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:29:16,878][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:29:17,384][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:29:17,884][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:29:18,387][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:29:18,893][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:29:19,391][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:29:19,891][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:29:20,387][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:29:20,885][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:29:21,388][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:29:21,888][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:29:22,386][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:29:22,897][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:29:23,394][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:29:23,891][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:29:24,392][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:29:24,889][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:29:25,390][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:29:25,890][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:29:26,387][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:29:26,884][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:29:27,383][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:29:27,884][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:29:28,384][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:29:28,885][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:29:29,391][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:29:29,887][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:29:30,385][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:29:30,884][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:29:31,382][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:29:31,882][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:29:32,376][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:29:32,871][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:29:33,367][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:29:33,863][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:29:34,358][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:29:34,852][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:29:35,349][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:29:35,852][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:29:36,352][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:29:36,851][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:29:37,351][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:29:37,848][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:29:38,350][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:29:38,846][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:29:39,344][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10133 tokens.
[2025-11-13 04:29:40,103][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 04:29:40,838][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:29:40,839][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:29:40,841][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:29:42,857][__main__][INFO] - Iteration 421 took 1m 5s (43.08% Gen, 53.84% Train). Generation: 28s, Training: 35s. Estimated remaining time: 48h 10m 0s. Estimated total time: 54h 30m 1s. Time estimates for 10 more iterations: 10m 54s, 100 more iterations: 1h 49m 0s, 500 more iterations: 9h 5m 0s.
[2025-11-13 04:29:42,859][__main__][INFO] - Starting iteration 421.
[2025-11-13 04:29:43,339][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:29:43,339][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:29:58,579][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:30:01,901][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:30:10,899][__main__][INFO] - Number of regex retries in iteration 421: 2
[2025-11-13 04:30:10,899][__main__][INFO] - agents played in iteration 421 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:30:11,817][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:30:11,841][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:30:11,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:30:11,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:30:11,887][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:30:11,888][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:30:12,763][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:30:13,226][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:30:13,735][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:30:14,239][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:30:14,743][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:30:15,245][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:30:15,747][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:30:16,254][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:30:16,762][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:30:17,268][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:30:17,770][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:30:18,274][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:30:18,776][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:30:19,282][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:30:19,784][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:30:20,290][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:30:20,796][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:30:21,296][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:30:21,797][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:30:22,300][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:30:22,804][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:30:23,309][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:30:23,811][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:30:24,324][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:30:24,826][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:30:25,331][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:30:25,849][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:30:26,350][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:30:26,853][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:30:27,359][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:30:27,861][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:30:28,365][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:30:28,867][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:30:29,366][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:30:29,866][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:30:30,364][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:30:30,861][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:30:31,362][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:30:31,858][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:30:32,362][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:30:32,861][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:30:33,362][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:30:33,865][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:30:34,364][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:30:34,861][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:30:35,359][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:30:35,855][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:30:36,355][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:30:36,852][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:30:37,351][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:30:37,861][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:30:38,360][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:30:38,863][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:30:39,362][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:30:39,863][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:30:40,365][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:30:40,863][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:30:41,361][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:30:41,864][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:30:42,366][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:30:42,866][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:30:43,371][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:30:43,868][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:30:44,366][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:30:44,865][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10106 tokens.
[2025-11-13 04:30:45,617][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.02%, ΔTime: 00:00:32
[2025-11-13 04:30:46,246][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:30:46,248][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:30:46,250][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:30:47,177][__main__][INFO] - Iteration 422 took 1m 3s (43.17% Gen, 55.37% Train). Generation: 27s, Training: 35s. Estimated remaining time: 46h 50m 49s. Estimated total time: 53h 11m 55s. Time estimates for 10 more iterations: 10m 38s, 100 more iterations: 1h 46m 23s, 500 more iterations: 8h 51m 59s.
[2025-11-13 04:30:47,179][__main__][INFO] - Starting iteration 422.
[2025-11-13 04:30:47,662][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:30:47,662][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:31:22,948][__main__][INFO] - Number of regex retries in iteration 422: 0
[2025-11-13 04:31:22,949][__main__][INFO] - agents played in iteration 422 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:31:23,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:31:23,830][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:31:23,860][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:31:23,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:31:23,886][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:31:23,886][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:31:24,663][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:31:25,119][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:31:25,625][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:31:26,130][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:31:26,643][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:31:27,143][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:31:27,643][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:31:28,149][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:31:28,650][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:31:29,164][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:31:29,663][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:31:30,161][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:31:30,662][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:31:31,163][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:31:31,663][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:31:32,161][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:31:32,661][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:31:33,182][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:31:33,686][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:31:34,189][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:31:34,691][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:31:35,192][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:31:35,698][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:31:36,199][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:31:36,700][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:31:37,201][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:31:37,699][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:31:38,198][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:31:38,698][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:31:39,197][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:31:39,695][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:31:40,197][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:31:40,698][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:31:41,204][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:31:41,706][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:31:42,209][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:31:42,709][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:31:43,206][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:31:43,705][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:31:44,204][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:31:44,703][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:31:45,203][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:31:45,699][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:31:46,195][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:31:46,694][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:31:47,197][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:31:47,693][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:31:48,189][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:31:48,686][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:31:49,187][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:31:49,686][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:31:50,186][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:31:50,686][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:31:51,184][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:31:51,684][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:31:52,182][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:31:52,682][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:31:53,185][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:31:53,683][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:31:54,186][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:31:54,690][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:31:55,195][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:31:55,696][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:31:56,197][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:31:56,700][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10093 tokens.
[2025-11-13 04:31:57,537][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 04:31:58,291][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:31:58,293][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:31:58,295][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:31:59,285][__main__][INFO] - Iteration 423 took 1m 11s (49.27% Gen, 49.35% Train). Generation: 35s, Training: 35s. Estimated remaining time: 53h 18m 55s. Estimated total time: 59h 41m 12s. Time estimates for 10 more iterations: 11m 56s, 100 more iterations: 1h 59m 22s, 500 more iterations: 9h 56m 52s.
[2025-11-13 04:31:59,287][__main__][INFO] - Starting iteration 423.
[2025-11-13 04:31:59,769][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:31:59,770][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:32:21,667][__main__][INFO] - Number of regex retries in iteration 423: 0
[2025-11-13 04:32:21,669][__main__][INFO] - agents played in iteration 423 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:32:22,503][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:32:22,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:32:22,555][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:32:22,578][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:32:22,578][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:32:22,579][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:32:23,417][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:32:23,876][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:32:24,385][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:32:24,891][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:32:25,394][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:32:25,900][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:32:26,401][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:32:26,905][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:32:27,413][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:32:27,920][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:32:28,425][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:32:28,927][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:32:29,429][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:32:29,936][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:32:30,438][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:32:30,940][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:32:31,456][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:32:31,956][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:32:32,473][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:32:32,972][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:32:33,470][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:32:33,977][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:32:34,481][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:32:34,983][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:32:35,485][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:32:35,988][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:32:36,493][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:32:36,996][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:32:37,497][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:32:38,000][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:32:38,511][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:32:39,014][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:32:39,516][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:32:40,018][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:32:40,521][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:32:41,024][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:32:41,525][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:32:42,027][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:32:42,531][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:32:43,033][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:32:43,532][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:32:44,031][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:32:44,533][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:32:45,034][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:32:45,545][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:32:46,045][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:32:46,546][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:32:47,067][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:32:47,568][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:32:48,065][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:32:48,564][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:32:49,063][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:32:49,568][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:32:50,070][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:32:50,572][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:32:51,073][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:32:51,574][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:32:52,078][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:32:52,577][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:32:53,081][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:32:53,586][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:32:54,086][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:32:54,589][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:32:55,092][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:32:55,595][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10120 tokens.
[2025-11-13 04:32:56,471][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:33
[2025-11-13 04:32:57,122][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:32:57,124][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:32:57,126][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:32:58,369][__main__][INFO] - Iteration 424 took 58s (37.37% Gen, 60.51% Train). Generation: 21s, Training: 35s. Estimated remaining time: 42h 26m 43s. Estimated total time: 48h 50m 0s. Time estimates for 10 more iterations: 9m 46s, 100 more iterations: 1h 37m 40s, 500 more iterations: 8h 8m 20s.
[2025-11-13 04:32:58,371][__main__][INFO] - Starting iteration 424.
[2025-11-13 04:32:58,865][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:32:58,866][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:33:33,582][__main__][INFO] - Number of regex retries in iteration 424: 0
[2025-11-13 04:33:33,583][__main__][INFO] - agents played in iteration 424 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:33:34,387][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:33:34,412][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:33:34,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:33:34,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:33:34,459][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:33:34,460][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:33:35,284][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:33:36,009][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:33:36,516][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:33:37,020][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:33:37,529][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:33:38,038][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:33:38,540][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:33:39,048][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:33:39,549][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:33:40,050][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:33:40,549][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:33:41,049][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:33:41,553][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:33:42,071][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:33:42,571][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:33:43,069][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:33:43,571][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:33:44,068][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:33:44,575][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:33:45,072][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:33:45,568][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:33:46,068][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:33:46,564][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:33:47,062][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:33:47,564][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:33:48,064][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:33:48,565][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:33:49,068][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:33:49,568][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:33:50,084][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:33:50,587][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:33:51,106][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:33:51,610][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:33:52,110][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:33:52,615][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:33:53,114][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:33:53,612][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:33:54,111][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:33:54,608][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:33:55,109][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:33:55,607][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:33:56,104][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:33:56,603][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:33:57,103][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:33:57,602][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:33:58,102][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:33:58,600][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:33:59,103][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:33:59,601][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:34:00,100][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:34:00,601][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:34:01,107][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:34:01,609][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:34:02,114][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:34:02,619][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:34:03,121][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:34:03,624][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:34:04,125][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:34:04,629][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:34:05,130][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:34:05,631][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:34:06,133][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:34:06,633][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:34:07,137][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:34:07,637][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10112 tokens.
[2025-11-13 04:34:08,466][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.37%, ΔTime: 00:00:33
[2025-11-13 04:34:09,219][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:34:09,221][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:34:09,224][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:34:10,246][__main__][INFO] - Iteration 425 took 1m 11s (48.63% Gen, 49.93% Train). Generation: 34s, Training: 35s. Estimated remaining time: 53h 4m 36s. Estimated total time: 59h 29m 5s. Time estimates for 10 more iterations: 11m 53s, 100 more iterations: 1h 58m 58s, 500 more iterations: 9h 54m 50s.
[2025-11-13 04:34:10,249][__main__][INFO] - Starting iteration 425.
[2025-11-13 04:34:10,717][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:34:10,718][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:34:31,970][__main__][INFO] - Number of regex retries in iteration 425: 0
[2025-11-13 04:34:31,972][__main__][INFO] - agents played in iteration 425 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:34:32,808][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:34:32,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:34:32,860][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:34:32,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:34:32,884][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:34:32,885][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:34:33,695][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:34:34,153][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:34:34,679][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:34:35,180][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:34:35,686][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:34:36,200][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:34:36,700][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:34:37,204][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:34:37,712][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:34:38,212][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:34:38,714][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:34:39,215][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:34:39,716][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:34:40,239][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:34:40,746][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:34:41,251][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:34:41,751][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:34:42,255][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:34:42,768][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:34:43,267][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:34:43,767][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:34:44,267][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:34:44,767][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:34:45,280][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:34:45,783][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:34:46,285][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:34:46,795][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:34:47,301][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:34:47,803][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:34:48,308][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:34:48,810][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:34:49,310][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:34:49,808][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:34:50,305][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:34:50,806][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:34:51,303][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:34:51,805][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:34:52,304][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:34:52,803][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:34:53,304][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:34:53,803][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:34:54,302][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:34:54,806][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:34:55,305][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:34:55,804][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:34:56,308][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:34:56,805][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:34:57,307][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:34:57,808][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:34:58,304][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:34:58,806][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:34:59,303][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:34:59,804][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:35:00,306][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:35:00,805][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:35:01,308][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:35:01,809][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:35:02,309][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:35:02,811][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:35:03,311][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:35:03,812][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:35:04,317][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:35:04,821][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:35:05,322][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:35:05,823][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10126 tokens.
[2025-11-13 04:35:06,647][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 04:35:07,304][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:35:07,306][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:35:07,308][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:35:08,234][__main__][INFO] - Iteration 426 took 57s (36.95% Gen, 61.44% Train). Generation: 21s, Training: 35s. Estimated remaining time: 41h 30m 24s. Estimated total time: 47h 55m 51s. Time estimates for 10 more iterations: 9m 35s, 100 more iterations: 1h 35m 51s, 500 more iterations: 7h 59m 18s.
[2025-11-13 04:35:08,236][__main__][INFO] - Starting iteration 426.
[2025-11-13 04:35:08,719][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:35:08,720][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:35:25,136][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:35:36,482][__main__][INFO] - Number of regex retries in iteration 426: 1
[2025-11-13 04:35:36,483][__main__][INFO] - agents played in iteration 426 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:35:37,291][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:35:37,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:35:37,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:35:37,369][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:35:37,369][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:35:37,371][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:35:38,176][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:35:38,638][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:35:39,144][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:35:39,649][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:35:40,148][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:35:40,650][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:35:41,153][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:35:41,655][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:35:42,167][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:35:42,672][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:35:43,182][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:35:43,694][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:35:44,199][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:35:44,708][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:35:45,212][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:35:45,718][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:35:46,222][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:35:46,728][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:35:47,228][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:35:47,729][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:35:48,231][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:35:48,734][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:35:49,235][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:35:49,738][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:35:50,241][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:35:50,742][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:35:51,245][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:35:51,747][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:35:52,248][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:35:52,754][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:35:53,256][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:35:53,761][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:35:54,264][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:35:54,764][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:35:55,281][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:35:55,786][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:35:56,287][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:35:56,786][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:35:57,287][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:35:57,794][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:35:58,291][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:35:58,790][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:35:59,289][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:35:59,788][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:36:00,287][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:36:00,786][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:36:01,285][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:36:01,794][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:36:02,293][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:36:02,791][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:36:03,292][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:36:03,790][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:36:04,292][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:36:04,790][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:36:05,288][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:36:05,804][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:36:06,303][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:36:06,803][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:36:07,305][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:36:07,806][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:36:08,308][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:36:08,809][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:36:09,312][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:36:09,811][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:36:10,312][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10121 tokens.
[2025-11-13 04:36:11,120][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 04:36:11,858][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:36:11,860][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:36:11,861][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:36:12,919][__main__][INFO] - Iteration 427 took 1m 4s (43.24% Gen, 55.11% Train). Generation: 27s, Training: 35s. Estimated remaining time: 47h 3m 29s. Estimated total time: 53h 30m 1s. Time estimates for 10 more iterations: 10m 42s, 100 more iterations: 1h 47m 0s, 500 more iterations: 8h 55m 0s.
[2025-11-13 04:36:12,921][__main__][INFO] - Starting iteration 427.
[2025-11-13 04:36:13,397][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:36:13,398][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:36:35,783][__main__][INFO] - Number of regex retries in iteration 427: 0
[2025-11-13 04:36:35,783][__main__][INFO] - agents played in iteration 427 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:36:36,571][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:36:36,595][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:36:36,618][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:36:36,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:36:36,641][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:36:36,641][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:36:37,392][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:36:37,849][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:36:38,351][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:36:38,850][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:36:39,349][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:36:39,848][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:36:40,345][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:36:40,844][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:36:41,345][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:36:41,848][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:36:42,351][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:36:42,857][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:36:43,357][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:36:43,858][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:36:44,358][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:36:44,861][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:36:45,366][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:36:45,870][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:36:46,373][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:36:46,881][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:36:47,382][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:36:47,898][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:36:48,400][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:36:48,904][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:36:49,411][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:36:49,911][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:36:50,416][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:36:50,919][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:36:51,423][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:36:53,070][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:36:53,968][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:36:54,499][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:36:55,001][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:36:55,505][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:36:56,008][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:36:56,512][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:36:57,018][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:36:57,524][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:36:58,027][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:36:58,534][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:36:59,034][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:36:59,534][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:37:00,035][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:37:00,537][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:37:01,044][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:37:01,544][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:37:02,043][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:37:02,546][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:37:03,046][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:37:03,546][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:37:04,044][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:37:04,543][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:37:05,043][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:37:05,540][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:37:06,038][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:37:06,536][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:37:07,034][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:37:07,539][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:37:08,043][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:37:08,543][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:37:09,046][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:37:09,551][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:37:10,049][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:37:10,552][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:37:11,053][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10096 tokens.
[2025-11-13 04:37:11,915][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.04%, ΔTime: 00:00:34
[2025-11-13 04:37:12,582][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:37:12,584][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:37:12,586][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:37:13,800][__main__][INFO] - Iteration 428 took 1m 0s (37.06% Gen, 60.93% Train). Generation: 22s, Training: 36s. Estimated remaining time: 43h 52m 37s. Estimated total time: 50h 20m 9s. Time estimates for 10 more iterations: 10m 4s, 100 more iterations: 1h 40m 40s, 500 more iterations: 8h 23m 21s.
[2025-11-13 04:37:13,802][__main__][INFO] - Starting iteration 428.
[2025-11-13 04:37:14,288][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:37:14,289][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:37:32,921][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:37:44,297][__main__][INFO] - Number of regex retries in iteration 428: 1
[2025-11-13 04:37:44,297][__main__][INFO] - agents played in iteration 428 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:37:45,075][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:37:45,104][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:37:45,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:37:45,154][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:37:45,155][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:37:45,156][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:37:45,897][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:37:46,354][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:37:46,864][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:37:47,369][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:37:47,871][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:37:48,388][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:37:48,886][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:37:49,394][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:37:49,899][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:37:50,401][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:37:50,916][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:37:51,418][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:37:51,918][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:37:52,427][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:37:52,934][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:37:53,435][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:37:53,938][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:37:54,439][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:37:54,945][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:37:55,448][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:37:56,011][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:37:56,516][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:37:57,019][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:37:57,524][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:37:58,027][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:37:58,529][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:37:59,036][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:37:59,543][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:38:00,045][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:38:00,548][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:38:01,052][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:38:01,555][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:38:02,055][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:38:02,556][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:38:03,061][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:38:03,563][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:38:04,065][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:38:04,570][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:38:05,074][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:38:05,614][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:38:06,118][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:38:06,616][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:38:07,113][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:38:07,609][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:38:08,118][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:38:08,614][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:38:09,112][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:38:09,630][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:38:10,131][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:38:10,629][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:38:11,126][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:38:11,625][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:38:12,127][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:38:12,626][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:38:13,124][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:38:13,630][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:38:14,135][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:38:14,637][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:38:15,138][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:38:15,639][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:38:16,145][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:38:16,649][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:38:17,153][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:38:17,654][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:38:18,159][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10104 tokens.
[2025-11-13 04:38:18,984][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:33
[2025-11-13 04:38:19,730][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:38:19,732][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:38:19,733][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:38:20,768][__main__][INFO] - Iteration 429 took 1m 6s (45.14% Gen, 53.30% Train). Generation: 30s, Training: 35s. Estimated remaining time: 48h 55m 21s. Estimated total time: 55h 24m 0s. Time estimates for 10 more iterations: 11m 4s, 100 more iterations: 1h 50m 48s, 500 more iterations: 9h 14m 0s.
[2025-11-13 04:38:20,770][__main__][INFO] - Starting iteration 429.
[2025-11-13 04:38:21,258][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:38:21,259][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:38:45,581][__main__][INFO] - Number of regex retries in iteration 429: 0
[2025-11-13 04:38:45,582][__main__][INFO] - agents played in iteration 429 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:38:46,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:38:46,402][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:38:46,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:38:46,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:38:46,452][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:38:46,453][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:38:47,225][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:38:47,685][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:38:48,191][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:38:48,695][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:38:49,199][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:38:49,696][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:38:50,193][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:38:50,697][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:38:51,203][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:38:51,708][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:38:52,208][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:38:52,711][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:38:53,230][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:38:53,729][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:38:54,231][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:38:54,733][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:38:55,234][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:38:55,735][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:38:56,240][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:38:56,746][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:38:57,253][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:38:57,754][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:38:58,259][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:38:58,766][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:38:59,266][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:38:59,766][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:39:00,272][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:39:02,128][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:39:02,709][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:39:03,211][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:39:03,726][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:39:04,230][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:39:04,730][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:39:05,231][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:39:05,732][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:39:06,241][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:39:06,743][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:39:07,245][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:39:07,751][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:39:08,261][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:39:08,763][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:39:09,266][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:39:09,768][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:39:10,269][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:39:10,772][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:39:11,275][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:39:11,775][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:39:12,273][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:39:12,774][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:39:13,276][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:39:13,778][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:39:14,289][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:39:14,802][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:39:15,313][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:39:15,828][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:39:16,340][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:39:16,844][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:39:17,346][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:39:17,849][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:39:18,360][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:39:18,863][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:39:19,365][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:39:19,864][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:39:20,367][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:39:20,875][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10125 tokens.
[2025-11-13 04:39:21,725][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:34
[2025-11-13 04:39:22,389][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:39:22,390][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:39:22,393][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:39:23,356][__main__][INFO] - Iteration 430 took 1m 2s (39.17% Gen, 59.28% Train). Generation: 24s, Training: 36s. Estimated remaining time: 45h 15m 12s. Estimated total time: 51h 44m 54s. Time estimates for 10 more iterations: 10m 20s, 100 more iterations: 1h 43m 29s, 500 more iterations: 8h 37m 29s.
[2025-11-13 04:39:23,358][__main__][INFO] - Starting iteration 430.
[2025-11-13 04:39:23,841][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 04:39:23,842][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:39:54,176][__main__][INFO] - Number of regex retries in iteration 430: 0
[2025-11-13 04:39:54,177][__main__][INFO] - agents played in iteration 430 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:39:55,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:39:55,067][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:39:55,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:39:55,122][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:39:55,123][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:39:55,124][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:39:55,853][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:39:56,311][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:39:56,827][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:39:57,324][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:39:57,842][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:39:58,339][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:39:58,837][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:39:59,354][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:39:59,855][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:40:00,354][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:40:00,856][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:40:01,356][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:40:01,863][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:40:02,365][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:40:02,870][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:40:03,377][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:40:03,884][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:40:04,382][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:40:04,883][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:40:05,385][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:40:05,887][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:40:06,390][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:40:06,891][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:40:07,395][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:40:07,897][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:40:08,401][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:40:08,904][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:40:09,407][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:40:09,921][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:40:10,428][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:40:10,931][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:40:11,434][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:40:11,936][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:40:12,456][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:40:12,956][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:40:13,459][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:40:13,964][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:40:14,464][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:40:14,967][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:40:15,470][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:40:15,972][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:40:16,480][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:40:16,979][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:40:17,481][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:40:17,981][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:40:18,479][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:40:18,985][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:40:19,485][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:40:19,985][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:40:20,485][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:40:20,983][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:40:21,486][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:40:21,985][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:40:22,483][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:40:22,985][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:40:23,485][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:40:23,987][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:40:24,493][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:40:24,998][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:40:25,502][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:40:26,006][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:40:26,509][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:40:27,027][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:40:27,531][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:40:28,036][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10188 tokens.
[2025-11-13 04:40:28,886][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:33
[2025-11-13 04:40:29,620][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:40:29,622][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:40:29,624][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:40:31,564][__main__][INFO] - Iteration 431 took 1m 7s (44.79% Gen, 52.34% Train). Generation: 30s, Training: 35s. Estimated remaining time: 49h 55m 21s. Estimated total time: 56h 26m 11s. Time estimates for 10 more iterations: 11m 17s, 100 more iterations: 1h 52m 52s, 500 more iterations: 9h 24m 21s.
[2025-11-13 04:40:31,567][__main__][INFO] - Starting iteration 431.
[2025-11-13 04:40:32,055][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:40:32,055][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:40:45,254][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:40:53,720][__main__][INFO] - Number of regex retries in iteration 431: 1
[2025-11-13 04:40:53,721][__main__][INFO] - agents played in iteration 431 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:40:54,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:40:54,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:40:54,542][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:40:54,565][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:40:54,565][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:40:54,566][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:40:55,281][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:40:55,857][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:40:56,370][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:40:56,868][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:40:57,366][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:40:57,873][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:40:58,373][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:40:58,878][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:40:59,379][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:40:59,878][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:41:00,374][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:41:00,869][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:41:01,374][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:41:01,869][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:41:02,368][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:41:02,864][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:41:03,359][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:41:03,862][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:41:04,358][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:41:04,855][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:41:05,351][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:41:05,847][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:41:06,343][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:41:06,841][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:41:07,337][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:41:07,856][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:41:08,360][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:41:08,861][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:41:09,363][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:41:09,865][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:41:10,364][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:41:10,865][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:41:11,372][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:41:12,679][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:41:13,461][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:41:13,961][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:41:14,472][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:41:14,972][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:41:15,475][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:41:15,975][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:41:16,477][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:41:16,985][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:41:17,486][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:41:17,991][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:41:18,499][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:41:19,003][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:41:19,508][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:41:20,011][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:41:20,520][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:41:21,023][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:41:21,524][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:41:22,029][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:41:22,540][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:41:23,043][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:41:23,549][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:41:24,054][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:41:24,567][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:41:25,075][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:41:25,578][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:41:26,086][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:41:26,592][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:41:27,097][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:41:27,604][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:41:28,109][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:41:28,612][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10170 tokens.
[2025-11-13 04:41:29,483][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:34
[2025-11-13 04:41:30,140][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:41:30,142][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:41:30,144][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:41:31,021][__main__][INFO] - Iteration 432 took 58s (36.74% Gen, 61.77% Train). Generation: 21s, Training: 36s. Estimated remaining time: 42h 36m 32s. Estimated total time: 49h 8m 21s. Time estimates for 10 more iterations: 9m 49s, 100 more iterations: 1h 38m 16s, 500 more iterations: 8h 11m 23s.
[2025-11-13 04:41:31,023][__main__][INFO] - Starting iteration 432.
[2025-11-13 04:41:31,515][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:41:31,516][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:41:49,376][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:42:01,693][__main__][INFO] - Number of regex retries in iteration 432: 1
[2025-11-13 04:42:01,694][__main__][INFO] - agents played in iteration 432 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:42:02,577][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:42:02,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:42:02,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:42:02,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:42:02,672][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:42:02,672][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:42:03,401][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:42:03,858][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:42:04,364][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:42:04,862][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:42:05,369][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:42:05,872][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:42:06,372][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:42:06,873][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:42:07,374][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:42:07,881][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:42:08,385][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:42:08,885][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:42:09,386][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:42:09,888][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:42:10,389][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:42:10,890][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:42:11,390][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:42:11,892][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:42:12,390][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:42:12,891][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:42:13,390][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:42:13,888][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:42:14,387][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:42:14,887][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:42:15,390][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:42:15,896][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:42:16,397][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:42:16,899][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:42:17,402][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:42:17,905][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:42:18,414][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:42:18,919][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:42:19,424][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:42:19,930][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:42:20,430][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:42:20,932][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:42:21,438][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:42:21,940][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:42:22,459][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:42:22,964][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:42:23,467][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:42:23,972][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:42:24,476][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:42:24,982][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:42:25,490][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:42:25,994][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:42:26,501][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:42:27,005][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:42:27,510][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:42:28,010][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:42:28,513][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:42:29,018][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:42:29,519][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:42:30,020][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:42:30,524][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:42:31,028][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:42:31,530][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:42:32,034][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:42:32,543][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:42:33,065][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:42:33,570][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:42:34,074][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:42:34,580][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:42:35,085][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:42:35,598][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10275 tokens.
[2025-11-13 04:42:36,450][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:33
[2025-11-13 04:42:37,188][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:42:37,190][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:42:37,191][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:42:38,201][__main__][INFO] - Iteration 433 took 1m 6s (45.25% Gen, 53.23% Train). Generation: 30s, Training: 35s. Estimated remaining time: 49h 1m 22s. Estimated total time: 55h 34m 19s. Time estimates for 10 more iterations: 11m 6s, 100 more iterations: 1h 51m 8s, 500 more iterations: 9h 15m 43s.
[2025-11-13 04:42:38,203][__main__][INFO] - Starting iteration 433.
[2025-11-13 04:42:38,686][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:42:38,687][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:42:50,461][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:43:02,197][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:43:02,984][__main__][INFO] - Number of regex retries in iteration 433: 2
[2025-11-13 04:43:02,984][__main__][INFO] - agents played in iteration 433 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:43:03,759][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:43:03,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:43:03,810][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:43:03,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:43:03,832][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:43:03,833][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:43:04,556][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:43:05,007][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:43:05,518][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:43:06,014][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:43:06,511][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:43:07,015][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:43:07,512][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:43:08,009][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:43:08,513][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:43:09,015][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:43:09,516][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:43:10,015][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:43:10,517][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:43:11,016][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:43:11,518][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:43:12,014][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:43:12,514][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:43:13,010][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:43:13,509][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:43:14,005][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:43:14,505][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:43:15,007][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:43:15,504][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:43:16,001][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:43:16,496][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:43:16,994][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:43:17,504][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:43:18,000][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:43:18,502][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:43:18,995][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:43:19,491][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:43:20,000][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:43:20,501][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:43:20,996][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:43:21,502][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:43:21,998][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:43:22,500][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:43:22,999][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:43:23,504][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:43:24,010][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:43:24,513][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:43:25,014][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:43:25,523][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:43:26,027][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:43:26,530][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:43:27,035][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:43:27,537][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:43:28,042][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:43:28,546][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:43:29,872][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:43:30,705][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:43:31,213][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:43:31,719][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:43:32,221][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:43:32,727][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:43:33,233][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:43:33,739][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:43:34,244][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:43:34,748][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:43:35,252][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:43:35,753][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:43:36,255][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:43:36,759][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:43:37,275][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:43:37,779][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10257 tokens.
[2025-11-13 04:43:38,673][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:34
[2025-11-13 04:43:39,317][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:43:39,319][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:43:39,320][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:43:40,218][__main__][INFO] - Iteration 434 took 1m 1s (39.49% Gen, 59.05% Train). Generation: 24s, Training: 36s. Estimated remaining time: 44h 42m 39s. Estimated total time: 51h 16m 38s. Time estimates for 10 more iterations: 10m 15s, 100 more iterations: 1h 42m 33s, 500 more iterations: 8h 32m 46s.
[2025-11-13 04:43:40,220][__main__][INFO] - Starting iteration 434.
[2025-11-13 04:43:40,702][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:43:40,702][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:44:09,908][__main__][INFO] - Number of regex retries in iteration 434: 0
[2025-11-13 04:44:09,909][__main__][INFO] - agents played in iteration 434 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:44:10,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:44:10,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:44:10,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:44:10,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:44:10,802][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:44:10,803][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:44:11,532][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:44:11,993][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:44:12,502][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:44:13,001][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:44:13,498][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:44:13,996][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:44:14,508][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:44:15,008][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:44:15,508][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:44:16,005][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:44:16,503][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:44:17,011][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:44:17,509][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:44:18,011][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:44:18,512][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:44:19,008][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:44:19,508][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:44:20,005][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:44:20,503][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:44:21,017][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:44:21,515][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:44:22,011][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:44:22,510][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:44:23,007][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:44:23,513][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:44:24,015][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:44:24,514][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:44:25,018][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:44:25,517][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:44:26,020][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:44:26,518][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:44:27,020][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:44:27,526][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:44:28,031][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:44:28,535][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:44:29,037][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:44:29,542][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:44:30,044][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:44:30,542][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:44:31,043][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:44:31,547][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:44:32,046][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:44:32,547][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:44:33,056][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:44:33,557][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:44:34,061][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:44:34,563][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:44:35,068][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:44:35,582][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:44:36,092][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:44:36,594][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:44:37,094][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:44:37,593][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:44:38,113][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:44:38,612][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:44:39,113][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:44:39,619][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:44:40,127][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:44:40,635][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:44:41,138][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:44:41,641][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:44:42,145][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:44:42,648][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:44:43,153][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:44:43,654][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10169 tokens.
[2025-11-13 04:44:44,510][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 04:44:45,236][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:44:45,238][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:44:45,239][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:44:46,459][__main__][INFO] - Iteration 435 took 1m 5s (44.41% Gen, 53.73% Train). Generation: 29s, Training: 35s. Estimated remaining time: 48h 12m 49s. Estimated total time: 54h 47m 54s. Time estimates for 10 more iterations: 10m 57s, 100 more iterations: 1h 49m 35s, 500 more iterations: 9h 7m 59s.
[2025-11-13 04:44:46,462][__main__][INFO] - Starting iteration 435.
[2025-11-13 04:44:46,948][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:44:46,949][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:45:12,609][__main__][INFO] - Number of regex retries in iteration 435: 0
[2025-11-13 04:45:12,610][__main__][INFO] - agents played in iteration 435 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:45:13,392][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:45:13,419][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:45:13,445][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:45:13,469][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:45:13,469][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:45:13,470][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:45:14,208][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:45:14,662][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:45:15,164][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:45:15,667][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:45:16,168][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:45:16,665][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:45:17,161][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:45:17,659][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:45:18,156][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:45:18,658][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:45:19,158][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:45:19,655][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:45:20,154][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:45:20,655][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:45:21,151][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:45:21,650][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:45:22,149][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:45:22,649][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:45:23,145][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:45:23,643][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:45:24,143][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:45:24,640][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:45:25,137][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:45:25,633][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:45:26,129][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:45:26,641][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:45:27,137][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:45:27,643][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:45:28,143][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:45:28,643][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:45:29,147][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:45:29,648][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:45:30,152][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:45:30,654][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:45:31,159][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:45:31,662][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:45:32,171][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:45:32,675][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:45:33,184][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:45:33,689][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:45:34,194][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:45:34,696][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:45:35,201][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:45:35,726][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:45:36,229][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:45:36,736][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:45:37,241][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:45:37,747][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:45:39,930][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:45:40,440][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:45:40,945][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:45:41,446][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:45:41,949][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:45:42,456][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:45:42,958][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:45:43,460][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:45:43,961][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:45:44,466][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:45:44,970][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:45:45,476][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:45:45,998][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:45:46,516][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:45:47,025][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:45:47,535][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:45:48,044][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10153 tokens.
[2025-11-13 04:45:48,914][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.37%, ΔTime: 00:00:34
[2025-11-13 04:45:49,592][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:45:49,595][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:45:49,597][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:45:50,516][__main__][INFO] - Iteration 436 took 1m 3s (40.37% Gen, 58.19% Train). Generation: 25s, Training: 36s. Estimated remaining time: 46h 22m 16s. Estimated total time: 52h 58m 25s. Time estimates for 10 more iterations: 10m 35s, 100 more iterations: 1h 45m 56s, 500 more iterations: 8h 49m 44s.
[2025-11-13 04:45:50,518][__main__][INFO] - Starting iteration 436.
[2025-11-13 04:45:51,000][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:45:51,001][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:46:17,232][__main__][INFO] - Number of regex retries in iteration 436: 0
[2025-11-13 04:46:17,233][__main__][INFO] - agents played in iteration 436 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:46:18,075][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:46:18,099][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:46:18,124][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:46:18,146][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:46:18,147][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:46:18,149][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:46:18,883][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:46:19,344][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:46:19,847][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:46:20,348][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:46:20,846][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:46:21,343][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:46:21,846][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:46:22,352][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:46:22,848][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:46:23,345][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:46:23,844][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:46:24,344][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:46:24,839][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:46:25,338][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:46:25,842][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:46:26,341][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:46:26,841][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:46:27,349][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:46:27,850][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:46:28,353][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:46:28,854][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:46:29,354][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:46:29,865][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:46:30,371][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:46:30,874][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:46:31,377][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:46:31,882][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:46:32,388][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:46:32,918][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:46:33,424][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:46:33,930][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:46:34,432][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:46:34,942][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:46:35,445][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:46:35,947][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:46:36,455][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:46:36,957][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:46:37,462][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:46:37,964][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:46:38,465][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:46:38,973][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:46:39,476][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:46:39,982][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:46:40,484][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:46:40,986][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:46:41,487][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:46:41,987][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:46:42,487][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:46:42,991][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:46:43,495][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:46:44,000][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:46:44,513][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:46:45,015][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:46:45,530][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:46:46,035][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:46:46,539][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:46:47,052][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:46:47,555][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:46:48,059][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:46:48,563][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:46:49,071][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:46:49,573][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:46:50,077][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:46:50,581][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:46:51,083][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10098 tokens.
[2025-11-13 04:46:51,942][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:33
[2025-11-13 04:46:52,694][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:46:52,695][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:46:52,697][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:46:53,890][__main__][INFO] - Iteration 437 took 1m 2s (41.71% Gen, 56.39% Train). Generation: 26s, Training: 35s. Estimated remaining time: 45h 47m 22s. Estimated total time: 52h 24m 34s. Time estimates for 10 more iterations: 10m 28s, 100 more iterations: 1h 44m 49s, 500 more iterations: 8h 44m 5s.
[2025-11-13 04:46:53,892][__main__][INFO] - Starting iteration 437.
[2025-11-13 04:46:54,375][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:46:54,375][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:47:21,672][__main__][INFO] - Number of regex retries in iteration 437: 0
[2025-11-13 04:47:21,672][__main__][INFO] - agents played in iteration 437 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:47:22,506][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:47:22,533][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:47:22,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:47:22,583][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:47:22,583][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:47:22,584][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:47:23,315][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:47:23,775][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:47:24,279][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:47:24,784][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:47:25,286][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:47:25,792][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:47:26,293][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:47:26,791][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:47:27,291][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:47:27,792][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:47:28,290][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:47:28,788][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:47:29,289][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:47:29,793][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:47:30,290][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:47:30,788][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:47:31,286][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:47:31,785][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:47:32,289][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:47:32,792][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:47:33,291][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:47:33,797][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:47:34,294][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:47:34,793][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:47:35,294][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:47:35,795][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:47:36,307][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:47:36,808][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:47:37,312][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:47:37,820][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:47:38,321][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:47:38,845][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:47:39,348][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:47:39,854][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:47:40,356][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:47:40,858][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:47:41,368][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:47:41,873][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:47:42,377][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:47:42,882][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:47:43,387][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:47:43,891][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:47:44,392][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:47:44,894][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:47:45,402][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:47:45,904][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:47:46,408][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:47:46,926][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:47:47,430][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:47:47,941][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:47:49,548][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:47:50,590][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:47:51,101][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:47:51,604][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:47:52,107][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:47:52,609][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:47:53,112][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:47:53,628][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:47:54,132][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:47:54,644][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:47:55,146][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:47:55,652][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:47:56,160][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:47:56,667][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:47:57,168][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10096 tokens.
[2025-11-13 04:47:58,027][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:34
[2025-11-13 04:47:58,679][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:47:58,681][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:47:58,683][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:47:59,582][__main__][INFO] - Iteration 438 took 1m 5s (41.86% Gen, 56.76% Train). Generation: 27s, Training: 37s. Estimated remaining time: 47h 42m 4s. Estimated total time: 54h 20m 22s. Time estimates for 10 more iterations: 10m 52s, 100 more iterations: 1h 48m 40s, 500 more iterations: 9h 3m 23s.
[2025-11-13 04:47:59,584][__main__][INFO] - Starting iteration 438.
[2025-11-13 04:48:00,093][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:48:00,094][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:48:34,530][__main__][INFO] - Number of regex retries in iteration 438: 0
[2025-11-13 04:48:34,530][__main__][INFO] - agents played in iteration 438 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:48:35,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:48:35,402][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:48:35,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:48:35,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:48:35,452][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:48:35,453][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:48:36,179][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:48:36,643][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:48:37,149][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:48:37,648][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:48:38,148][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:48:38,650][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:48:39,154][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:48:39,654][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:48:40,152][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:48:40,651][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:48:41,150][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:48:41,650][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:48:42,149][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:48:42,646][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:48:43,158][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:48:43,656][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:48:44,155][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:48:44,661][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:48:45,165][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:48:45,666][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:48:46,170][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:48:46,676][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:48:47,179][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:48:47,682][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:48:48,188][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:48:48,689][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:48:49,191][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:48:49,696][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:48:50,199][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:48:50,700][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:48:51,207][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:48:51,708][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:48:52,210][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:48:52,713][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:48:53,215][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:48:53,734][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:48:54,237][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:48:54,746][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:48:55,250][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:48:55,821][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:48:56,338][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:48:56,844][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:48:57,347][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:48:57,855][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:48:58,367][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:48:58,871][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:48:59,375][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:48:59,878][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:49:00,385][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:49:00,887][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:49:01,388][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:49:01,894][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:49:02,396][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:49:02,895][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:49:03,399][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:49:03,902][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:49:04,405][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:49:04,909][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:49:05,411][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:49:05,918][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:49:06,418][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:49:06,923][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:49:07,426][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:49:07,930][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:49:08,458][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10093 tokens.
[2025-11-13 04:49:09,356][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 04:49:10,139][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:49:10,141][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:49:10,144][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:49:11,239][__main__][INFO] - Iteration 439 took 1m 11s (48.40% Gen, 50.06% Train). Generation: 34s, Training: 35s. Estimated remaining time: 52h 37m 53s. Estimated total time: 59h 17m 22s. Time estimates for 10 more iterations: 11m 51s, 100 more iterations: 1h 58m 34s, 500 more iterations: 9h 52m 53s.
[2025-11-13 04:49:11,241][__main__][INFO] - Starting iteration 439.
[2025-11-13 04:49:11,713][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:49:11,713][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:49:31,088][__main__][INFO] - Number of regex retries in iteration 439: 0
[2025-11-13 04:49:31,089][__main__][INFO] - agents played in iteration 439 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:49:31,969][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:49:31,996][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:49:32,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:49:32,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:49:32,045][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:49:32,046][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:49:32,774][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:49:33,227][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:49:33,728][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:49:34,230][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:49:34,757][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:49:35,254][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:49:35,750][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:49:36,247][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:49:36,748][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:49:37,255][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:49:37,757][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:49:38,255][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:49:38,764][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:49:39,265][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:49:39,765][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:49:40,271][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:49:40,771][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:49:41,272][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:49:41,769][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:49:42,267][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:49:42,771][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:49:43,271][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:49:43,774][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:49:44,280][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:49:44,785][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:49:45,286][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:49:45,790][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:49:46,294][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:49:46,796][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:49:47,295][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:49:47,798][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:49:48,303][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:49:48,806][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:49:49,310][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:49:49,811][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:49:50,319][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:49:50,822][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:49:51,326][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:49:51,832][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:49:52,337][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:49:52,842][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:49:53,355][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:49:53,861][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:49:54,367][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:49:54,872][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:49:55,379][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:49:55,886][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:49:56,388][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:49:56,891][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:49:57,393][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:49:57,918][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:49:58,430][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:49:58,932][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:49:59,437][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:49:59,944][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:50:02,114][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:50:02,629][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:50:03,130][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:50:03,635][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:50:04,140][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:50:04,642][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:50:05,160][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:50:05,662][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:50:06,162][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:50:06,663][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10317 tokens.
[2025-11-13 04:50:07,543][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:34
[2025-11-13 04:50:08,206][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:50:08,207][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:50:08,209][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:50:09,125][__main__][INFO] - Iteration 440 took 57s (33.75% Gen, 64.66% Train). Generation: 19s, Training: 37s. Estimated remaining time: 41h 10m 11s. Estimated total time: 47h 50m 38s. Time estimates for 10 more iterations: 9m 34s, 100 more iterations: 1h 35m 41s, 500 more iterations: 7h 58m 26s.
[2025-11-13 04:50:09,127][__main__][INFO] - Starting iteration 440.
[2025-11-13 04:50:09,604][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 04:50:09,605][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:50:42,352][__main__][INFO] - Number of regex retries in iteration 440: 0
[2025-11-13 04:50:42,353][__main__][INFO] - agents played in iteration 440 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:50:43,285][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:50:43,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:50:43,338][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:50:43,360][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:50:43,361][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:50:43,361][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:50:44,095][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:50:44,565][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:50:45,073][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:50:45,570][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:50:46,069][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:50:46,570][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:50:47,076][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:50:47,578][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:50:48,077][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:50:48,577][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:50:49,077][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:50:49,580][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:50:50,079][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:50:50,581][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:50:51,105][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:50:51,606][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:50:52,111][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:50:52,621][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:50:53,126][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:50:53,634][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:50:54,141][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:50:54,649][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:50:55,155][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:50:55,660][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:50:56,169][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:50:56,677][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:50:57,182][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:50:57,691][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:50:58,194][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:50:58,701][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:50:59,210][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:50:59,716][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:51:00,226][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:51:00,727][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:51:01,230][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:51:01,739][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:51:02,241][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:51:02,750][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:51:03,251][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:51:03,754][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:51:04,271][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:51:04,778][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:51:05,283][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:51:05,783][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:51:06,289][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:51:06,795][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:51:07,302][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:51:07,809][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:51:08,313][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:51:08,813][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:51:09,313][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:51:09,818][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:51:10,320][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:51:10,829][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:51:11,332][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:51:11,835][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:51:12,350][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:51:12,849][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:51:13,359][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:51:13,859][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:51:14,361][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:51:14,865][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:51:15,366][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:51:15,868][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:51:16,367][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10232 tokens.
[2025-11-13 04:51:17,220][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.45%, ΔTime: 00:00:33
[2025-11-13 04:51:17,977][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:51:17,979][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:51:17,981][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:51:19,920][__main__][INFO] - Iteration 441 took 1m 10s (46.57% Gen, 50.67% Train). Generation: 32s, Training: 35s. Estimated remaining time: 51h 54m 10s. Estimated total time: 58h 35m 48s. Time estimates for 10 more iterations: 11m 43s, 100 more iterations: 1h 57m 11s, 500 more iterations: 9h 45m 58s.
[2025-11-13 04:51:19,923][__main__][INFO] - Starting iteration 441.
[2025-11-13 04:51:20,403][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 04:51:20,403][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:51:49,176][__main__][INFO] - Number of regex retries in iteration 441: 0
[2025-11-13 04:51:49,177][__main__][INFO] - agents played in iteration 441 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:51:49,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:51:49,992][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:51:50,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:51:50,043][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:51:50,044][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:51:50,046][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:51:50,787][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:51:51,248][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:51:51,748][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:51:52,247][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:51:52,751][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:51:53,248][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:51:53,748][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:51:54,245][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:51:54,746][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:51:55,245][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:51:55,740][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:51:56,235][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:51:56,732][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:51:57,228][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:51:57,726][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:51:58,223][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:51:58,722][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:51:59,227][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:51:59,728][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:52:00,232][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:52:00,735][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:52:01,238][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:52:01,742][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:52:02,247][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:52:02,749][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:52:03,265][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:52:03,768][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:52:04,270][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:52:04,777][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:52:05,280][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:52:05,798][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:52:06,309][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:52:06,811][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:52:07,319][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:52:07,823][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:52:08,329][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:52:08,835][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:52:09,337][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:52:09,846][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:52:10,352][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:52:10,856][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:52:11,357][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:52:11,858][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:52:12,377][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:52:12,878][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:52:13,378][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:52:15,434][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:52:16,118][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:52:16,626][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:52:17,129][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:52:17,633][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:52:18,134][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:52:18,634][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:52:19,135][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:52:19,635][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:52:20,135][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:52:20,638][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:52:21,140][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:52:21,645][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:52:22,146][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:52:22,649][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:52:23,157][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:52:23,661][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:52:24,164][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:52:24,668][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10127 tokens.
[2025-11-13 04:52:25,589][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:34
[2025-11-13 04:52:26,266][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:52:26,268][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:52:26,270][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:52:27,480][__main__][INFO] - Iteration 442 took 1m 7s (42.89% Gen, 55.30% Train). Generation: 28s, Training: 37s. Estimated remaining time: 49h 11m 7s. Estimated total time: 55h 53m 53s. Time estimates for 10 more iterations: 11m 10s, 100 more iterations: 1h 51m 47s, 500 more iterations: 9h 18m 58s.
[2025-11-13 04:52:27,483][__main__][INFO] - Starting iteration 442.
[2025-11-13 04:52:27,968][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 04:52:27,968][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:53:00,623][__main__][INFO] - Number of regex retries in iteration 442: 0
[2025-11-13 04:53:00,624][__main__][INFO] - agents played in iteration 442 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:53:01,477][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:53:01,505][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:53:01,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:53:01,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:53:01,555][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:53:01,556][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:53:02,338][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:53:02,794][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:53:03,306][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:53:03,806][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:53:04,310][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:53:04,811][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:53:05,320][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:53:05,822][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:53:06,330][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:53:06,833][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:53:07,337][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:53:07,840][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:53:08,342][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:53:08,848][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:53:09,354][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:53:09,859][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:53:10,364][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:53:10,865][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:53:11,367][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:53:11,880][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:53:12,380][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:53:12,886][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:53:13,389][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:53:13,892][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:53:14,394][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:53:14,896][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:53:15,399][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:53:15,906][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:53:16,409][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:53:16,912][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:53:17,414][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:53:17,924][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:53:18,430][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:53:18,936][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:53:19,438][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:53:19,944][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:53:20,449][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:53:20,950][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:53:21,455][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:53:21,962][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:53:22,467][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:53:22,967][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:53:23,472][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:53:23,973][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:53:24,475][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:53:24,980][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:53:25,484][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:53:25,989][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:53:26,494][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:53:26,994][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:53:27,512][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:53:28,012][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:53:28,514][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:53:29,017][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:53:29,520][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:53:30,021][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:53:30,523][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:53:31,031][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:53:31,538][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:53:32,043][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:53:32,546][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:53:33,047][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:53:33,549][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:53:34,057][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:53:34,564][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10249 tokens.
[2025-11-13 04:53:35,425][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:33
[2025-11-13 04:53:36,180][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:53:36,182][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:53:36,184][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:53:37,169][__main__][INFO] - Iteration 443 took 1m 9s (47.19% Gen, 51.39% Train). Generation: 32s, Training: 35s. Estimated remaining time: 50h 56m 11s. Estimated total time: 57h 40m 7s. Time estimates for 10 more iterations: 11m 32s, 100 more iterations: 1h 55m 20s, 500 more iterations: 9h 36m 41s.
[2025-11-13 04:53:37,171][__main__][INFO] - Starting iteration 443.
[2025-11-13 04:53:37,650][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 04:53:37,651][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:53:54,045][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:53:54,302][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:54:06,678][__main__][INFO] - Number of regex retries in iteration 443: 2
[2025-11-13 04:54:06,679][__main__][INFO] - agents played in iteration 443 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:54:07,482][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:54:07,505][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:54:07,527][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:54:07,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:54:07,551][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:54:07,551][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:54:08,407][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:54:08,864][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:54:09,369][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:54:09,873][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:54:10,374][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:54:10,881][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:54:11,382][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:54:11,887][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:54:12,403][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:54:12,908][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:54:13,423][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:54:13,929][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:54:14,432][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:54:14,940][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:54:15,442][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:54:15,951][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:54:16,454][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:54:16,960][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:54:17,488][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:54:17,995][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:54:18,499][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:54:19,009][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:54:19,513][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:54:20,024][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:54:20,530][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:54:21,035][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:54:21,540][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:54:22,048][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:54:22,554][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:54:23,054][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:54:23,557][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:54:24,060][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:54:24,565][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:54:26,722][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:54:27,255][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:54:27,755][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:54:28,255][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:54:28,759][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:54:29,262][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:54:29,760][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:54:30,260][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:54:30,763][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:54:31,265][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:54:31,769][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:54:32,271][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:54:32,773][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:54:33,281][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:54:33,781][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:54:34,284][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:54:34,786][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:54:35,293][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:54:35,796][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:54:36,296][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:54:36,795][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:54:37,298][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:54:37,806][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:54:38,311][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:54:38,815][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:54:39,316][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:54:39,838][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:54:40,343][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:54:40,847][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:54:41,352][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:54:41,854][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:54:42,360][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10229 tokens.
[2025-11-13 04:54:43,220][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.43%, ΔTime: 00:00:34
[2025-11-13 04:54:43,903][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:54:43,906][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:54:43,908][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:54:45,134][__main__][INFO] - Iteration 444 took 1m 7s (43.01% Gen, 55.17% Train). Generation: 29s, Training: 37s. Estimated remaining time: 49h 29m 9s. Estimated total time: 56h 14m 13s. Time estimates for 10 more iterations: 11m 14s, 100 more iterations: 1h 52m 28s, 500 more iterations: 9h 22m 22s.
[2025-11-13 04:54:45,137][__main__][INFO] - Starting iteration 444.
[2025-11-13 04:54:45,615][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 04:54:45,615][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:55:18,121][__main__][INFO] - Number of regex retries in iteration 444: 0
[2025-11-13 04:55:18,121][__main__][INFO] - agents played in iteration 444 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:55:18,993][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:55:19,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:55:19,043][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:55:19,067][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:55:19,067][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:55:19,068][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:55:19,904][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:55:20,365][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:55:20,871][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:55:21,375][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:55:21,879][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:55:22,379][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:55:22,879][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:55:23,379][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:55:23,890][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:55:24,394][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:55:24,900][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:55:25,400][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:55:25,915][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:55:26,418][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:55:26,950][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:55:27,455][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:55:27,960][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:55:28,470][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:55:28,975][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:55:29,479][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:55:29,983][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:55:30,487][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:55:30,995][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:55:31,499][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:55:32,006][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:55:32,517][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:55:33,017][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:55:33,523][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:55:34,025][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:55:34,526][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:55:35,043][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:55:35,547][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:55:36,047][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:55:36,552][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:55:37,054][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:55:37,555][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:55:38,061][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:55:38,559][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:55:39,060][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:55:39,559][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:55:40,063][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:55:40,566][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:55:41,064][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:55:41,565][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:55:42,069][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:55:42,572][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:55:43,070][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:55:43,571][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:55:44,073][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:55:44,576][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:55:45,077][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:55:45,582][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:55:46,082][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:55:46,586][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:55:47,092][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:55:47,597][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:55:48,099][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:55:48,604][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:55:49,105][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:55:49,618][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:55:50,127][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:55:50,639][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:55:51,141][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:55:51,645][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:55:52,145][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10136 tokens.
[2025-11-13 04:55:52,975][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:33
[2025-11-13 04:55:53,731][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:55:53,733][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:55:53,734][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:55:55,070][__main__][INFO] - Iteration 445 took 1m 9s (46.80% Gen, 51.28% Train). Generation: 32s, Training: 35s. Estimated remaining time: 51h 6m 34s. Estimated total time: 57h 52m 47s. Time estimates for 10 more iterations: 11m 34s, 100 more iterations: 1h 55m 45s, 500 more iterations: 9h 38m 47s.
[2025-11-13 04:55:55,073][__main__][INFO] - Starting iteration 445.
[2025-11-13 04:55:55,687][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 04:55:55,688][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:56:19,100][__main__][INFO] - Number of regex retries in iteration 445: 0
[2025-11-13 04:56:19,101][__main__][INFO] - agents played in iteration 445 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:56:19,874][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:56:19,902][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:56:19,927][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:56:19,950][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:56:19,951][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:56:19,952][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:56:20,690][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:56:21,146][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:56:21,650][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:56:22,165][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:56:22,666][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:56:23,172][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:56:23,678][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:56:24,178][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:56:24,679][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:56:25,180][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:56:25,683][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:56:26,185][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:56:26,689][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:56:27,191][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:56:27,694][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:56:28,195][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:56:28,698][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:56:29,198][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:56:29,700][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:56:30,203][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:56:30,705][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:56:31,206][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:56:31,708][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:56:32,210][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:56:32,712][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:56:33,220][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:56:33,726][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:56:34,241][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:56:34,746][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:56:35,260][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:56:35,765][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:56:36,268][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:56:36,777][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:56:37,283][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:56:39,014][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:56:39,642][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:56:40,140][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:56:40,661][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:56:41,162][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:56:41,661][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:56:42,168][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:56:42,671][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:56:43,176][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:56:43,685][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:56:44,187][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:56:44,692][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:56:45,194][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:56:45,695][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:56:46,199][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:56:46,702][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:56:47,205][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:56:47,707][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:56:48,211][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:56:48,715][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:56:49,218][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:56:49,723][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:56:50,225][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:56:50,730][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:56:51,246][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:56:51,750][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:56:52,250][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:56:52,751][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:56:53,254][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:56:53,761][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:56:54,267][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10104 tokens.
[2025-11-13 04:56:55,130][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:34
[2025-11-13 04:56:55,779][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:56:55,781][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:56:55,783][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:56:56,885][__main__][INFO] - Iteration 446 took 1m 1s (38.26% Gen, 59.94% Train). Generation: 23s, Training: 36s. Estimated remaining time: 44h 12m 39s. Estimated total time: 50h 59m 54s. Time estimates for 10 more iterations: 10m 11s, 100 more iterations: 1h 41m 59s, 500 more iterations: 8h 29m 59s.
[2025-11-13 04:56:56,888][__main__][INFO] - Starting iteration 446.
[2025-11-13 04:56:57,357][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 04:56:57,358][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:57:16,919][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 04:57:26,316][__main__][INFO] - Number of regex retries in iteration 446: 1
[2025-11-13 04:57:26,317][__main__][INFO] - agents played in iteration 446 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:57:27,128][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:57:27,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:57:27,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:57:27,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:57:27,203][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:57:27,204][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:57:28,051][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:57:28,511][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:57:29,017][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:57:29,522][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:57:30,024][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:57:30,526][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:57:31,028][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:57:31,534][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:57:32,049][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:57:32,550][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:57:33,055][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:57:33,558][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:57:34,064][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:57:34,583][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:57:35,084][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:57:35,587][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:57:36,088][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:57:36,591][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:57:37,096][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:57:37,604][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:57:38,105][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:57:38,608][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:57:39,116][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:57:39,621][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:57:40,126][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:57:40,630][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:57:41,132][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:57:41,633][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:57:42,134][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:57:42,634][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:57:43,136][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:57:43,637][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:57:44,136][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:57:44,636][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:57:45,152][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:57:45,656][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:57:46,162][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:57:46,668][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:57:47,172][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:57:47,686][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:57:48,191][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:57:48,692][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:57:49,197][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:57:49,698][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:57:50,198][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:57:50,696][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:57:51,198][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:57:51,699][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:57:52,207][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:57:52,708][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:57:53,210][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:57:53,711][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:57:54,214][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:57:54,714][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:57:55,214][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:57:55,718][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:57:56,221][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:57:56,724][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:57:57,233][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:57:57,737][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:57:58,239][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:57:58,744][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:57:59,246][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:57:59,773][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:58:00,277][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10048 tokens.
[2025-11-13 04:58:01,085][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.08%, ΔTime: 00:00:33
[2025-11-13 04:58:01,818][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:58:01,820][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:58:01,821][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:58:02,850][__main__][INFO] - Iteration 447 took 1m 5s (44.22% Gen, 54.21% Train). Generation: 28s, Training: 35s. Estimated remaining time: 47h 46m 20s. Estimated total time: 54h 34m 41s. Time estimates for 10 more iterations: 10m 54s, 100 more iterations: 1h 49m 9s, 500 more iterations: 9h 5m 46s.
[2025-11-13 04:58:02,852][__main__][INFO] - Starting iteration 447.
[2025-11-13 04:58:03,343][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 04:58:03,343][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:58:27,850][__main__][INFO] - Number of regex retries in iteration 447: 0
[2025-11-13 04:58:27,850][__main__][INFO] - agents played in iteration 447 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:58:28,656][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:58:28,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:58:28,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:58:28,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:58:28,733][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:58:28,734][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:58:29,460][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:58:29,914][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:58:30,427][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:58:30,929][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:58:31,437][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:58:31,937][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:58:32,439][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:58:32,942][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:58:33,445][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:58:33,952][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:58:34,452][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:58:34,952][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:58:35,465][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:58:35,971][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:58:36,488][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:58:36,989][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:58:37,491][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:58:37,994][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:58:38,499][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:58:39,007][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:58:39,511][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:58:40,012][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:58:40,514][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:58:41,016][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:58:41,522][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:58:42,027][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:58:42,532][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:58:43,040][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:58:43,543][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:58:44,047][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:58:44,559][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:58:45,065][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:58:45,573][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:58:46,078][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:58:46,577][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:58:47,093][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:58:47,594][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 04:58:48,094][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 04:58:49,907][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 04:58:50,963][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 04:58:51,464][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 04:58:51,966][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 04:58:52,490][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 04:58:52,991][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 04:58:53,493][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 04:58:54,001][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 04:58:54,503][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 04:58:55,011][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 04:58:55,516][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 04:58:56,018][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 04:58:56,521][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 04:58:57,025][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 04:58:57,527][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 04:58:58,036][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 04:58:58,557][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 04:58:59,070][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 04:58:59,579][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 04:59:00,086][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 04:59:00,591][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 04:59:01,095][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 04:59:01,605][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 04:59:02,108][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 04:59:02,616][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 04:59:03,123][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 04:59:03,625][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10218 tokens.
[2025-11-13 04:59:04,532][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:35
[2025-11-13 04:59:05,180][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 04:59:05,183][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 04:59:05,185][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 04:59:06,165][__main__][INFO] - Iteration 448 took 1m 2s (39.01% Gen, 59.43% Train). Generation: 24s, Training: 37s. Estimated remaining time: 45h 31m 43s. Estimated total time: 52h 21m 7s. Time estimates for 10 more iterations: 10m 28s, 100 more iterations: 1h 44m 42s, 500 more iterations: 8h 43m 31s.
[2025-11-13 04:59:06,167][__main__][INFO] - Starting iteration 448.
[2025-11-13 04:59:06,654][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 04:59:06,654][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 04:59:39,830][__main__][INFO] - Number of regex retries in iteration 448: 0
[2025-11-13 04:59:39,831][__main__][INFO] - agents played in iteration 448 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 04:59:40,688][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:59:40,724][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:59:40,748][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:59:40,771][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 04:59:40,772][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 04:59:40,774][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 04:59:41,607][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 04:59:42,066][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 04:59:42,577][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 04:59:43,080][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 04:59:43,593][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 04:59:44,098][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 04:59:44,607][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 04:59:45,113][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 04:59:45,622][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 04:59:46,126][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 04:59:46,632][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 04:59:47,144][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 04:59:47,653][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 04:59:48,155][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 04:59:48,657][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 04:59:49,163][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 04:59:49,667][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 04:59:50,176][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 04:59:50,677][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 04:59:51,178][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 04:59:51,682][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 04:59:52,186][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 04:59:52,696][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 04:59:53,201][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 04:59:53,706][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 04:59:54,212][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 04:59:54,713][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 04:59:55,219][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 04:59:55,728][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 04:59:56,233][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 04:59:56,744][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 04:59:57,245][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 04:59:57,747][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 04:59:58,254][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 04:59:58,755][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 04:59:59,256][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 04:59:59,767][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:00:00,272][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:00:00,791][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:00:01,297][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:00:01,804][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:00:02,310][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:00:02,814][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:00:03,320][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:00:03,826][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:00:04,328][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:00:04,851][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:00:05,359][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:00:05,862][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:00:06,361][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:00:06,866][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:00:07,393][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:00:07,896][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:00:08,397][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:00:08,899][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:00:09,401][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:00:09,908][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:00:10,405][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:00:10,907][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:00:11,409][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:00:11,908][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:00:12,409][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:00:12,912][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:00:13,413][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:00:13,914][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10225 tokens.
[2025-11-13 05:00:14,700][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.37%, ΔTime: 00:00:33
[2025-11-13 05:00:15,454][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:00:15,456][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:00:15,457][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:00:16,548][__main__][INFO] - Iteration 449 took 1m 9s (47.47% Gen, 50.97% Train). Generation: 33s, Training: 35s. Estimated remaining time: 51h 24m 8s. Estimated total time: 58h 14m 43s. Time estimates for 10 more iterations: 11m 38s, 100 more iterations: 1h 56m 29s, 500 more iterations: 9h 42m 27s.
[2025-11-13 05:00:16,550][__main__][INFO] - Starting iteration 449.
[2025-11-13 05:00:17,018][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 05:00:17,019][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:00:43,484][__main__][INFO] - Number of regex retries in iteration 449: 0
[2025-11-13 05:00:43,485][__main__][INFO] - agents played in iteration 449 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:00:44,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:00:44,345][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:00:44,372][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:00:44,395][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:00:44,396][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:00:44,397][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:00:45,187][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:00:45,651][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:00:46,157][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:00:46,660][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:00:47,176][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:00:47,682][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:00:48,202][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:00:48,706][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:00:49,207][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:00:49,716][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:00:50,223][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:00:50,729][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:00:51,231][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:00:51,733][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:00:52,240][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:00:52,741][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:00:53,248][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:00:53,751][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:00:55,426][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:00:56,379][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:00:56,885][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:00:57,399][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:00:57,899][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:00:58,403][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:00:58,908][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:00:59,410][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:00:59,915][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:01:00,414][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:01:00,924][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:01:01,428][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:01:01,930][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:01:02,462][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:01:02,966][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:01:03,470][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:01:03,971][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:01:04,481][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:01:04,982][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:01:05,506][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:01:06,013][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:01:06,515][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:01:07,020][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:01:07,521][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:01:08,025][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:01:08,525][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:01:09,028][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:01:09,530][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:01:10,032][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:01:10,533][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:01:11,032][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:01:11,536][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:01:12,037][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:01:12,544][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:01:13,046][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:01:13,555][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:01:14,055][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:01:14,559][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:01:15,057][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:01:15,558][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:01:16,063][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:01:16,565][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:01:17,065][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:01:17,569][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:01:18,069][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:01:18,572][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:01:19,075][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10207 tokens.
[2025-11-13 05:01:19,959][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:34
[2025-11-13 05:01:20,603][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:01:20,605][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:01:20,607][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:01:21,565][__main__][INFO] - Iteration 450 took 1m 4s (41.00% Gen, 57.51% Train). Generation: 26s, Training: 37s. Estimated remaining time: 46h 55m 42s. Estimated total time: 53h 47m 22s. Time estimates for 10 more iterations: 10m 45s, 100 more iterations: 1h 47m 34s, 500 more iterations: 8h 57m 53s.
[2025-11-13 05:01:21,567][__main__][INFO] - Starting iteration 450.
[2025-11-13 05:01:22,048][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 05:01:22,049][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:01:57,836][__main__][INFO] - Number of regex retries in iteration 450: 0
[2025-11-13 05:01:57,837][__main__][INFO] - agents played in iteration 450 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:01:58,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:01:58,691][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:01:58,716][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:01:58,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:01:58,739][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:01:58,740][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:01:59,618][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:02:00,082][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:02:00,591][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:02:01,108][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:02:01,611][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:02:02,116][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:02:02,630][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:02:03,136][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:02:03,640][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:02:04,139][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:02:04,641][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:02:05,147][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:02:05,649][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:02:06,149][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:02:06,648][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:02:07,148][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:02:07,652][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:02:08,152][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:02:08,652][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:02:09,151][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:02:09,652][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:02:10,155][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:02:10,655][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:02:11,156][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:02:11,658][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:02:12,160][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:02:12,664][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:02:13,168][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:02:13,668][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:02:14,173][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:02:14,677][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:02:15,180][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:02:15,694][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:02:16,195][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:02:16,710][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:02:17,219][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:02:17,721][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:02:18,226][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:02:18,729][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:02:19,231][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:02:19,735][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:02:20,233][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:02:20,743][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:02:21,244][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:02:21,743][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:02:22,244][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:02:22,744][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:02:23,244][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:02:23,747][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:02:24,251][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:02:24,754][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:02:25,257][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:02:25,755][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:02:26,256][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:02:26,753][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:02:27,251][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:02:27,749][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:02:28,251][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:02:28,748][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:02:29,251][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:02:29,751][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:02:30,255][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:02:30,754][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:02:31,252][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:02:31,753][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10141 tokens.
[2025-11-13 05:02:32,543][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:32
[2025-11-13 05:02:33,282][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:02:33,283][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:02:33,284][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:02:35,569][__main__][INFO] - Iteration 451 took 1m 13s (48.68% Gen, 48.21% Train). Generation: 35s, Training: 35s. Estimated remaining time: 54h 23m 10s. Estimated total time: 61h 16m 4s. Time estimates for 10 more iterations: 12m 15s, 100 more iterations: 2h 2m 32s, 500 more iterations: 10h 12m 40s.
[2025-11-13 05:02:35,571][__main__][INFO] - Starting iteration 451.
[2025-11-13 05:02:36,044][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:02:36,045][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:03:00,072][__main__][INFO] - Number of regex retries in iteration 451: 0
[2025-11-13 05:03:00,075][__main__][INFO] - agents played in iteration 451 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:03:00,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:03:01,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:03:01,038][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:03:01,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:03:01,062][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:03:01,063][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:03:01,956][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:03:02,419][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:03:02,934][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:03:03,436][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:03:03,939][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:03:04,445][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:03:04,951][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:03:05,453][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:03:05,956][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:03:06,458][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:03:06,959][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:03:07,462][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:03:07,963][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:03:08,462][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:03:08,966][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:03:09,468][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:03:09,972][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:03:10,474][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:03:10,974][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:03:11,474][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:03:11,976][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:03:12,476][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:03:12,975][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:03:13,477][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:03:13,976][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:03:14,476][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:03:14,976][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:03:15,477][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:03:15,980][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:03:16,481][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:03:16,982][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:03:17,483][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:03:17,984][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:03:18,486][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:03:18,989][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:03:19,492][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:03:19,999][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:03:20,504][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:03:21,008][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:03:21,539][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:03:22,041][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:03:22,549][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:03:23,053][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:03:23,557][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:03:24,060][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:03:24,566][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:03:25,073][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:03:25,575][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:03:26,074][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:03:26,584][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:03:27,090][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:03:27,596][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:03:28,108][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:03:28,610][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:03:29,118][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:03:29,625][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:03:30,128][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:03:30,632][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:03:31,135][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:03:31,636][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:03:32,138][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:03:32,642][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:03:33,142][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:03:33,641][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:03:34,145][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10052 tokens.
[2025-11-13 05:03:34,974][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:33
[2025-11-13 05:03:35,624][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:03:35,626][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:03:35,628][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:03:36,519][__main__][INFO] - Iteration 452 took 1m 0s (39.74% Gen, 58.79% Train). Generation: 24s, Training: 35s. Estimated remaining time: 43h 29m 50s. Estimated total time: 50h 23m 44s. Time estimates for 10 more iterations: 10m 4s, 100 more iterations: 1h 40m 47s, 500 more iterations: 8h 23m 57s.
[2025-11-13 05:03:36,521][__main__][INFO] - Starting iteration 452.
[2025-11-13 05:03:37,009][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:03:37,009][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:04:10,218][__main__][INFO] - Number of regex retries in iteration 452: 0
[2025-11-13 05:04:10,218][__main__][INFO] - agents played in iteration 452 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:04:11,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:04:11,092][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:04:11,116][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:04:11,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:04:11,142][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:04:11,142][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:04:11,996][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:04:12,458][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:04:12,967][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:04:13,469][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:04:13,974][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:04:14,476][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:04:14,980][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:04:15,487][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:04:15,995][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:04:16,494][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:04:16,997][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:04:17,496][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:04:17,999][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:04:18,501][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:04:19,004][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:04:19,514][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:04:20,013][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:04:20,516][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:04:21,016][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:04:21,514][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:04:22,027][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:04:22,529][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:04:23,029][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:04:23,534][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:04:24,032][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:04:24,552][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:04:25,051][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:04:25,554][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:04:26,054][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:04:26,558][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:04:27,063][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:04:27,566][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:04:28,068][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:04:28,574][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:04:29,079][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:04:29,583][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:04:30,086][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:04:30,592][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:04:31,099][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:04:31,604][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:04:32,109][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:04:32,622][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:04:33,124][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:04:33,632][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:04:34,134][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:04:34,638][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:04:35,154][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:04:35,663][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:04:36,177][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:04:36,679][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:04:37,185][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:04:37,692][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:04:38,195][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:04:38,697][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:04:39,204][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:04:39,709][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:04:40,211][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:04:40,714][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:04:41,218][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:04:41,723][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:04:42,222][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:04:42,722][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:04:43,231][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:04:43,732][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:04:44,255][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10089 tokens.
[2025-11-13 05:04:45,035][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:33
[2025-11-13 05:04:45,788][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:04:45,790][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:04:45,791][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:04:46,813][__main__][INFO] - Iteration 453 took 1m 9s (47.57% Gen, 50.96% Train). Generation: 33s, Training: 35s. Estimated remaining time: 51h 15m 9s. Estimated total time: 58h 10m 14s. Time estimates for 10 more iterations: 11m 38s, 100 more iterations: 1h 56m 20s, 500 more iterations: 9h 41m 42s.
[2025-11-13 05:04:46,815][__main__][INFO] - Starting iteration 453.
[2025-11-13 05:04:47,286][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:04:47,286][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:05:07,322][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:05:09,909][__main__][INFO] - Number of regex retries in iteration 453: 1
[2025-11-13 05:05:09,910][__main__][INFO] - agents played in iteration 453 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:05:12,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:05:12,811][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:05:12,834][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:05:12,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:05:12,857][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:05:12,858][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:05:13,708][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:05:14,173][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:05:14,682][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:05:15,189][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:05:15,698][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:05:16,202][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:05:16,708][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:05:17,219][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:05:17,726][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:05:18,234][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:05:18,736][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:05:19,240][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:05:19,748][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:05:20,252][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:05:20,754][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:05:21,260][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:05:21,766][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:05:22,274][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:05:22,775][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:05:23,276][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:05:23,775][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:05:24,279][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:05:24,781][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:05:25,283][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:05:25,783][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:05:26,286][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:05:26,788][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:05:27,291][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:05:27,794][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:05:28,303][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:05:28,807][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:05:29,308][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:05:29,812][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:05:30,321][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:05:30,826][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:05:31,330][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:05:31,834][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:05:32,337][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:05:32,858][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:05:33,369][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:05:33,874][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:05:34,379][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:05:34,880][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:05:35,381][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:05:35,882][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:05:36,383][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:05:36,882][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:05:37,384][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:05:37,887][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:05:38,387][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:05:38,894][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:05:39,396][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:05:39,900][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:05:40,404][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:05:40,904][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:05:41,408][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:05:41,910][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:05:42,411][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:05:42,914][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:05:43,420][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:05:43,920][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:05:44,420][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:05:44,921][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:05:45,421][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:05:45,926][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10119 tokens.
[2025-11-13 05:05:46,781][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:33
[2025-11-13 05:05:47,443][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:05:47,445][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:05:47,447][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:05:48,429][__main__][INFO] - Iteration 454 took 1m 1s (37.00% Gen, 61.39% Train). Generation: 22s, Training: 37s. Estimated remaining time: 44h 1m 5s. Estimated total time: 50h 57m 12s. Time estimates for 10 more iterations: 10m 11s, 100 more iterations: 1h 41m 54s, 500 more iterations: 8h 29m 32s.
[2025-11-13 05:05:48,431][__main__][INFO] - Starting iteration 454.
[2025-11-13 05:05:48,913][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:05:48,914][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:06:21,800][__main__][INFO] - Number of regex retries in iteration 454: 0
[2025-11-13 05:06:21,800][__main__][INFO] - agents played in iteration 454 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:06:22,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:06:22,693][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:06:22,717][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:06:22,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:06:22,741][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:06:22,742][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:06:23,594][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:06:24,065][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:06:24,573][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:06:25,078][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:06:25,588][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:06:26,092][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:06:26,592][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:06:27,093][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:06:27,594][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:06:28,107][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:06:28,610][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:06:29,113][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:06:29,614][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:06:30,115][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:06:30,619][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:06:31,117][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:06:31,622][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:06:32,125][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:06:32,629][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:06:33,133][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:06:33,635][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:06:34,136][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:06:34,640][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:06:35,143][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:06:35,644][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:06:36,159][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:06:36,668][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:06:37,181][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:06:37,681][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:06:38,182][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:06:38,683][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:06:39,183][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:06:39,682][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:06:40,183][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:06:40,683][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:06:41,192][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:06:41,694][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:06:42,196][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:06:42,698][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:06:43,202][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:06:43,704][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:06:44,218][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:06:44,730][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:06:45,245][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:06:45,758][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:06:46,269][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:06:46,772][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:06:47,273][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:06:47,776][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:06:48,285][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:06:48,787][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:06:49,291][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:06:49,795][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:06:50,299][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:06:50,799][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:06:51,302][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:06:51,806][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:06:52,308][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:06:52,809][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:06:53,314][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:06:53,819][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:06:54,323][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:06:54,828][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:06:55,328][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:06:55,829][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10110 tokens.
[2025-11-13 05:06:56,626][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:33
[2025-11-13 05:06:57,358][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:06:57,359][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:06:57,361][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:06:58,352][__main__][INFO] - Iteration 455 took 1m 9s (47.36% Gen, 51.21% Train). Generation: 32s, Training: 35s. Estimated remaining time: 50h 54m 41s. Estimated total time: 57h 51m 58s. Time estimates for 10 more iterations: 11m 34s, 100 more iterations: 1h 55m 43s, 500 more iterations: 9h 38m 39s.
[2025-11-13 05:06:58,354][__main__][INFO] - Starting iteration 455.
[2025-11-13 05:06:58,848][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:06:58,848][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:07:21,199][__main__][INFO] - Number of regex retries in iteration 455: 0
[2025-11-13 05:07:21,200][__main__][INFO] - agents played in iteration 455 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:07:23,701][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:07:23,724][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:07:23,746][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:07:23,769][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:07:23,770][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:07:23,770][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:07:24,641][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:07:25,104][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:07:25,612][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:07:26,115][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:07:26,622][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:07:27,129][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:07:27,634][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:07:28,140][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:07:28,648][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:07:29,150][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:07:29,652][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:07:30,152][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:07:30,654][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:07:31,164][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:07:31,667][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:07:32,174][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:07:32,685][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:07:33,186][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:07:33,699][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:07:34,200][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:07:34,708][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:07:35,213][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:07:35,720][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:07:36,220][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:07:36,725][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:07:37,230][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:07:37,736][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:07:38,242][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:07:38,745][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:07:39,249][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:07:39,754][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:07:40,262][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:07:40,769][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:07:41,271][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:07:41,780][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:07:42,285][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:07:42,800][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:07:43,301][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:07:43,814][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:07:44,316][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:07:44,817][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:07:45,320][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:07:45,824][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:07:46,328][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:07:46,839][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:07:47,343][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:07:47,847][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:07:48,352][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:07:48,855][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:07:49,377][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:07:49,879][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:07:50,381][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:07:50,885][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:07:51,386][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:07:51,890][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:07:52,392][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:07:52,894][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:07:53,395][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:07:53,898][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:07:54,399][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:07:54,904][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:07:55,405][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:07:55,912][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:07:56,413][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:07:56,917][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10086 tokens.
[2025-11-13 05:07:57,776][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:33
[2025-11-13 05:07:58,430][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:07:58,433][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:07:58,435][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:07:59,376][__main__][INFO] - Iteration 456 took 1m 0s (36.93% Gen, 61.52% Train). Generation: 22s, Training: 37s. Estimated remaining time: 43h 28m 7s. Estimated total time: 50h 26m 25s. Time estimates for 10 more iterations: 10m 5s, 100 more iterations: 1h 40m 52s, 500 more iterations: 8h 24m 24s.
[2025-11-13 05:07:59,377][__main__][INFO] - Starting iteration 456.
[2025-11-13 05:07:59,880][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:07:59,881][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:08:31,133][__main__][INFO] - Number of regex retries in iteration 456: 0
[2025-11-13 05:08:31,133][__main__][INFO] - agents played in iteration 456 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:08:31,969][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:08:31,992][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:08:32,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:08:32,038][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:08:32,038][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:08:32,039][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:08:32,890][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:08:33,351][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:08:33,857][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:08:34,362][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:08:34,862][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:08:35,364][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:08:35,877][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:08:36,381][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:08:36,898][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:08:37,406][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:08:37,909][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:08:38,416][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:08:38,918][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:08:39,420][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:08:39,921][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:08:40,420][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:08:40,926][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:08:41,424][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:08:41,925][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:08:42,429][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:08:42,928][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:08:43,428][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:08:43,929][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:08:44,428][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:08:44,937][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:08:45,439][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:08:45,944][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:08:46,448][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:08:46,957][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:08:47,476][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:08:47,978][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:08:48,483][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:08:48,986][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:08:49,485][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:08:49,986][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:08:50,484][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:08:50,981][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:08:51,494][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:08:51,994][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:08:52,493][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:08:52,994][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:08:53,494][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:08:54,002][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:08:54,502][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:08:55,005][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:08:55,511][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:08:56,017][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:08:56,524][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:08:57,028][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:08:57,531][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:08:58,031][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:08:58,535][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:08:59,037][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:08:59,538][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:09:00,039][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:09:00,542][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:09:01,043][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:09:01,546][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:09:02,050][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:09:02,555][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:09:03,057][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:09:03,555][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:09:04,058][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:09:04,559][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:09:05,057][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10138 tokens.
[2025-11-13 05:09:05,915][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 05:09:06,644][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:09:06,645][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:09:06,647][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:09:07,773][__main__][INFO] - Iteration 457 took 1m 7s (46.03% Gen, 52.31% Train). Generation: 31s, Training: 35s. Estimated remaining time: 49h 35m 15s. Estimated total time: 56h 34m 41s. Time estimates for 10 more iterations: 11m 18s, 100 more iterations: 1h 53m 9s, 500 more iterations: 9h 25m 46s.
[2025-11-13 05:09:07,775][__main__][INFO] - Starting iteration 457.
[2025-11-13 05:09:08,257][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:09:08,258][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:09:23,439][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:09:38,027][__main__][INFO] - Number of regex retries in iteration 457: 1
[2025-11-13 05:09:38,029][__main__][INFO] - agents played in iteration 457 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:09:38,937][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:09:38,978][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:09:39,011][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:09:39,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:09:39,039][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:09:39,040][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:09:39,894][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:09:40,355][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:09:40,863][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:09:41,371][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:09:41,874][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:09:42,385][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:09:42,889][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:09:43,395][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:09:43,903][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:09:44,405][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:09:44,915][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:09:45,418][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:09:45,918][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:09:46,435][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:09:46,939][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:09:47,440][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:09:47,942][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:09:48,445][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:09:48,949][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:09:49,455][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:09:49,957][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:09:50,459][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:09:50,966][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:09:51,470][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:09:51,973][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:09:52,471][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:09:52,983][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:09:53,482][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:09:53,983][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:09:54,492][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:09:54,994][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:09:55,512][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:09:56,014][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:09:56,517][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:09:57,017][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:09:57,521][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:09:58,028][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:09:58,531][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:09:59,035][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:09:59,537][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:10:00,038][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:10:00,538][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:10:01,039][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:10:01,542][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:10:02,042][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:10:02,542][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:10:03,045][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:10:03,548][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:10:04,048][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:10:04,548][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:10:05,048][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:10:05,550][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:10:06,053][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:10:06,555][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:10:07,058][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:10:07,557][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:10:08,061][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:10:08,563][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:10:09,063][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:10:09,568][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:10:10,074][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:10:10,577][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:10:11,081][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:10:11,583][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:10:12,087][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10160 tokens.
[2025-11-13 05:10:12,951][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:33
[2025-11-13 05:10:13,616][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:10:13,618][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:10:13,620][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:10:14,595][__main__][INFO] - Iteration 458 took 1m 6s (44.88% Gen, 53.65% Train). Generation: 29s, Training: 35s. Estimated remaining time: 48h 16m 22s. Estimated total time: 55h 16m 55s. Time estimates for 10 more iterations: 11m 3s, 100 more iterations: 1h 50m 33s, 500 more iterations: 9h 12m 49s.
[2025-11-13 05:10:14,597][__main__][INFO] - Starting iteration 458.
[2025-11-13 05:10:15,102][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:10:15,103][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:10:36,059][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:10:42,481][mllm.models.large_language_model_local][WARNING] - Response Propagation: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:10:48,566][__main__][INFO] - Number of regex retries in iteration 458: 2
[2025-11-13 05:10:48,567][__main__][INFO] - agents played in iteration 458 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:10:49,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:10:49,404][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:10:49,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:10:49,455][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:10:49,455][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:10:49,456][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:10:50,246][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:10:50,707][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:10:51,213][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:10:51,718][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:10:52,225][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:10:52,730][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:10:53,237][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:10:53,738][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:10:54,250][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:10:54,751][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:10:55,255][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:10:55,757][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:10:56,259][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:10:56,758][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:10:57,258][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:10:57,756][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:10:58,258][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:10:58,760][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:10:59,267][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:10:59,767][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:11:00,270][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:11:00,774][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:11:01,274][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:11:01,771][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:11:02,276][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:11:02,775][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:11:03,272][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:11:03,770][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:11:04,269][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:11:04,778][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:11:05,281][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:11:05,779][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:11:06,283][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:11:06,788][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:11:07,287][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:11:07,789][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:11:08,288][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:11:08,791][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:11:09,292][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:11:09,796][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:11:10,299][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:11:10,801][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:11:11,312][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:11:11,812][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:11:12,316][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:11:12,823][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:11:13,327][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:11:13,837][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:11:14,340][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:11:14,843][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:11:15,344][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:11:15,843][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:11:16,344][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:11:16,849][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:11:17,352][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:11:17,858][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:11:18,363][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:11:18,865][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:11:19,365][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:11:19,866][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:11:20,369][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:11:20,869][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:11:21,365][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:11:21,865][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:11:22,361][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10160 tokens.
[2025-11-13 05:11:23,166][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 05:11:23,901][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:11:23,903][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:11:23,905][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:11:25,011][__main__][INFO] - Iteration 459 took 1m 9s (47.87% Gen, 50.55% Train). Generation: 33s, Training: 35s. Estimated remaining time: 51h 13m 46s. Estimated total time: 58h 15m 29s. Time estimates for 10 more iterations: 11m 39s, 100 more iterations: 1h 56m 30s, 500 more iterations: 9h 42m 34s.
[2025-11-13 05:11:25,013][__main__][INFO] - Starting iteration 459.
[2025-11-13 05:11:25,485][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:11:25,486][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:11:41,732][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:11:46,734][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:11:53,613][__main__][INFO] - Number of regex retries in iteration 459: 2
[2025-11-13 05:11:53,614][__main__][INFO] - agents played in iteration 459 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:11:54,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:11:54,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:11:54,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:11:54,594][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:11:54,594][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:11:54,595][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:11:55,417][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:11:55,961][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:11:56,480][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:11:56,986][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:11:57,487][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:11:57,987][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:11:58,489][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:11:59,000][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:11:59,500][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:12:00,000][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:12:00,500][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:12:01,001][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:12:01,503][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:12:02,003][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:12:02,506][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:12:03,006][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:12:03,507][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:12:04,021][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:12:04,520][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:12:05,021][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:12:05,523][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:12:06,028][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:12:06,533][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:12:07,033][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:12:07,533][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:12:08,042][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:12:08,542][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:12:09,048][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:12:09,547][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:12:10,047][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:12:10,551][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:12:11,050][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:12:11,550][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:12:12,056][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:12:12,558][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:12:13,058][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:12:13,558][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:12:14,059][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:12:14,563][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:12:15,063][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:12:15,566][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:12:16,065][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:12:16,567][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:12:17,069][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:12:17,570][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:12:18,071][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:12:18,575][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:12:19,075][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:12:19,575][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:12:20,081][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:12:20,582][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:12:21,086][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:12:21,590][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:12:22,093][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:12:22,607][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:12:23,108][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:12:23,616][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:12:24,120][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:12:24,621][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:12:25,135][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:12:25,640][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:12:26,141][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:12:26,644][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:12:27,146][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:12:27,654][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10083 tokens.
[2025-11-13 05:12:28,525][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:33
[2025-11-13 05:12:29,211][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:12:29,213][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:12:29,214][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:12:30,196][__main__][INFO] - Iteration 460 took 1m 4s (43.47% Gen, 55.01% Train). Generation: 28s, Training: 35s. Estimated remaining time: 46h 52m 47s. Estimated total time: 53h 55m 35s. Time estimates for 10 more iterations: 10m 47s, 100 more iterations: 1h 47m 51s, 500 more iterations: 8h 59m 15s.
[2025-11-13 05:12:30,198][__main__][INFO] - Starting iteration 460.
[2025-11-13 05:12:30,678][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 05:12:30,678][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:12:52,214][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:13:01,144][__main__][INFO] - Number of regex retries in iteration 460: 1
[2025-11-13 05:13:01,144][__main__][INFO] - agents played in iteration 460 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:13:01,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:13:01,948][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:13:01,972][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:13:01,995][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:13:01,995][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:13:01,996][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:13:02,763][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:13:03,224][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:13:03,735][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:13:04,238][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:13:04,756][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:13:05,258][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:13:05,759][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:13:06,269][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:13:06,776][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:13:07,278][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:13:07,778][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:13:08,279][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:13:08,785][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:13:09,285][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:13:09,787][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:13:10,292][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:13:10,793][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:13:11,297][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:13:11,798][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:13:12,300][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:13:12,812][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:13:13,312][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:13:13,820][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:13:14,323][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:13:14,828][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:13:15,345][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:13:15,843][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:13:16,339][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:13:16,837][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:13:17,337][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:13:17,854][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:13:18,350][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:13:18,847][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:13:19,358][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:13:19,864][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:13:20,365][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:13:20,865][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:13:21,367][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:13:21,872][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:13:22,378][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:13:22,878][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:13:23,380][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:13:23,879][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:13:24,385][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:13:24,885][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:13:25,388][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:13:25,892][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:13:26,392][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:13:26,897][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:13:27,404][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:13:27,904][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:13:28,406][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:13:28,909][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:13:29,412][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:13:29,915][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:13:30,415][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:13:30,918][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:13:31,422][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:13:31,921][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:13:32,446][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:13:32,946][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:13:33,446][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:13:33,946][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:13:34,446][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:13:34,953][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10158 tokens.
[2025-11-13 05:13:36,063][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:33
[2025-11-13 05:13:36,818][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:13:36,820][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:13:36,821][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:13:38,741][__main__][INFO] - Iteration 461 took 1m 8s (44.76% Gen, 52.42% Train). Generation: 30s, Training: 35s. Estimated remaining time: 49h 39m 14s. Estimated total time: 56h 43m 11s. Time estimates for 10 more iterations: 11m 20s, 100 more iterations: 1h 53m 26s, 500 more iterations: 9h 27m 11s.
[2025-11-13 05:13:38,743][__main__][INFO] - Starting iteration 461.
[2025-11-13 05:13:39,208][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:13:39,209][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:14:05,818][__main__][INFO] - Number of regex retries in iteration 461: 0
[2025-11-13 05:14:05,821][__main__][INFO] - agents played in iteration 461 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:14:06,668][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:14:06,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:14:06,716][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:14:06,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:14:06,740][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:14:06,741][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:14:07,549][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:14:08,025][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:14:08,536][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:14:09,035][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:14:09,539][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:14:10,042][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:14:10,556][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:14:11,057][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:14:11,557][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:14:12,059][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:14:12,569][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:14:13,072][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:14:13,574][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:14:14,078][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:14:14,579][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:14:15,080][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:14:15,586][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:14:16,096][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:14:16,598][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:14:17,105][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:14:17,606][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:14:18,110][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:14:18,612][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:14:19,114][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:14:19,622][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:14:20,124][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:14:20,624][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:14:21,136][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:14:21,637][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:14:22,142][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:14:22,645][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:14:23,144][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:14:23,647][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:14:24,149][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:14:24,651][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:14:25,155][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:14:25,658][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:14:26,162][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:14:26,663][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:14:27,165][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:14:27,674][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:14:28,175][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:14:28,681][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:14:29,181][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:14:29,686][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:14:30,191][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:14:30,694][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:14:31,197][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:14:31,701][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:14:32,200][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:14:32,700][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:14:33,199][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:14:33,698][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:14:34,200][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:14:34,702][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:14:35,203][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:14:35,703][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:14:36,207][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:14:36,711][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:14:37,212][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:14:37,714][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:14:38,216][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:14:38,716][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:14:39,217][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:14:39,732][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10102 tokens.
[2025-11-13 05:14:40,610][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 05:14:41,266][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:14:41,268][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:14:41,270][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:14:42,227][__main__][INFO] - Iteration 462 took 1m 3s (42.23% Gen, 56.25% Train). Generation: 26s, Training: 35s. Estimated remaining time: 45h 25m 56s. Estimated total time: 52h 30m 57s. Time estimates for 10 more iterations: 10m 30s, 100 more iterations: 1h 45m 1s, 500 more iterations: 8h 45m 9s.
[2025-11-13 05:14:42,229][__main__][INFO] - Starting iteration 462.
[2025-11-13 05:14:42,699][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:14:42,699][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:15:01,281][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:15:13,577][__main__][INFO] - Number of regex retries in iteration 462: 1
[2025-11-13 05:15:13,578][__main__][INFO] - agents played in iteration 462 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:15:14,372][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:15:14,397][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:15:14,422][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:15:14,445][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:15:14,446][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:15:14,447][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:15:15,257][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:15:15,712][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:15:16,260][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:15:16,759][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:15:17,259][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:15:17,766][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:15:18,268][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:15:18,778][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:15:19,279][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:15:19,781][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:15:20,306][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:15:20,812][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:15:21,319][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:15:21,832][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:15:22,333][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:15:22,852][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:15:23,351][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:15:23,853][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:15:24,355][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:15:24,854][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:15:25,363][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:15:25,873][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:15:26,372][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:15:26,875][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:15:27,376][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:15:27,877][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:15:28,376][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:15:28,877][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:15:29,379][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:15:29,877][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:15:30,376][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:15:30,879][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:15:31,375][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:15:31,875][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:15:32,376][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:15:32,873][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:15:33,383][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:15:33,885][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:15:34,384][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:15:34,882][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:15:35,381][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:15:35,888][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:15:36,391][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:15:36,893][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:15:37,399][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:15:37,924][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:15:38,430][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:15:38,934][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:15:39,439][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:15:39,955][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:15:40,460][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:15:40,970][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:15:41,471][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:15:41,971][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:15:42,480][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:15:42,985][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:15:43,484][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:15:43,985][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:15:44,489][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:15:44,991][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:15:45,494][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:15:46,010][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:15:48,219][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:15:48,808][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:15:49,312][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10164 tokens.
[2025-11-13 05:15:50,184][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:34
[2025-11-13 05:15:50,866][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:15:50,868][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:15:50,870][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:15:51,817][__main__][INFO] - Iteration 463 took 1m 9s (44.67% Gen, 53.95% Train). Generation: 30s, Training: 37s. Estimated remaining time: 50h 29m 47s. Estimated total time: 57h 35m 57s. Time estimates for 10 more iterations: 11m 31s, 100 more iterations: 1h 55m 11s, 500 more iterations: 9h 35m 59s.
[2025-11-13 05:15:51,821][__main__][INFO] - Starting iteration 463.
[2025-11-13 05:15:52,299][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:15:52,299][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:16:17,311][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:16:23,881][__main__][INFO] - Number of regex retries in iteration 463: 1
[2025-11-13 05:16:23,882][__main__][INFO] - agents played in iteration 463 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:16:24,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:16:24,766][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:16:24,792][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:16:24,815][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:16:24,816][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:16:24,816][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:16:25,663][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:16:26,121][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:16:26,634][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:16:27,134][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:16:27,650][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:16:28,152][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:16:28,653][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:16:29,160][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:16:29,664][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:16:30,183][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:16:30,683][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:16:31,185][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:16:31,688][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:16:32,189][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:16:32,690][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:16:33,191][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:16:33,691][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:16:34,194][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:16:34,694][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:16:35,194][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:16:35,694][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:16:36,204][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:16:36,703][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:16:37,204][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:16:37,702][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:16:38,211][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:16:38,710][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:16:39,211][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:16:39,713][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:16:40,213][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:16:40,716][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:16:41,217][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:16:41,715][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:16:42,213][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:16:42,714][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:16:43,215][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:16:43,715][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:16:44,213][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:16:44,712][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:16:45,211][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:16:45,709][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:16:46,210][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:16:46,708][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:16:47,209][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:16:47,710][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:16:48,212][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:16:48,713][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:16:49,212][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:16:49,713][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:16:50,211][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:16:50,711][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:16:51,214][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:16:51,716][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:16:52,217][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:16:52,718][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:16:53,219][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:16:53,723][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:16:54,222][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:16:54,722][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:16:55,229][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:16:55,730][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:16:56,233][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:16:56,733][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:16:57,235][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:16:57,750][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10038 tokens.
[2025-11-13 05:16:58,571][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 05:16:59,361][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:16:59,363][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:16:59,364][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:17:00,444][__main__][INFO] - Iteration 464 took 1m 8s (46.35% Gen, 52.07% Train). Generation: 31s, Training: 35s. Estimated remaining time: 49h 39m 57s. Estimated total time: 56h 47m 16s. Time estimates for 10 more iterations: 11m 21s, 100 more iterations: 1h 53m 34s, 500 more iterations: 9h 27m 52s.
[2025-11-13 05:17:00,446][__main__][INFO] - Starting iteration 464.
[2025-11-13 05:17:00,924][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:17:00,924][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:17:16,348][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:17:27,352][__main__][INFO] - Number of regex retries in iteration 464: 1
[2025-11-13 05:17:27,353][__main__][INFO] - agents played in iteration 464 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:17:28,137][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:17:28,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:17:28,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:17:28,209][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:17:28,210][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:17:28,211][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:17:29,053][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:17:29,509][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:17:30,020][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:17:30,520][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:17:31,021][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:17:31,524][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:17:32,026][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:17:32,527][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:17:33,030][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:17:33,531][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:17:34,031][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:17:34,534][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:17:35,031][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:17:35,533][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:17:36,032][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:17:36,531][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:17:37,029][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:17:37,528][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:17:38,031][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:17:38,545][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:17:39,048][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:17:39,557][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:17:40,063][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:17:40,564][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:17:41,066][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:17:41,564][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:17:42,061][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:17:42,558][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:17:43,055][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:17:43,562][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:17:44,063][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:17:44,559][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:17:45,060][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:17:45,559][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:17:46,056][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:17:46,553][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:17:47,048][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:17:47,558][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:17:48,054][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:17:48,549][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:17:49,055][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:17:49,554][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:17:50,060][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:17:50,559][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:17:51,057][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:17:51,556][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:17:52,052][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:17:52,551][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:17:53,054][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:17:53,555][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:17:54,068][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:17:54,567][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:17:55,067][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:17:55,586][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:17:56,122][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:17:57,637][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:17:58,505][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:17:59,010][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:17:59,512][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:18:00,013][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:18:00,513][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:18:01,016][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:18:01,517][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:18:02,021][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:18:02,529][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10043 tokens.
[2025-11-13 05:18:03,407][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:34
[2025-11-13 05:18:04,090][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:18:04,091][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:18:04,093][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:18:04,999][__main__][INFO] - Iteration 465 took 1m 4s (41.24% Gen, 57.34% Train). Generation: 26s, Training: 36s. Estimated remaining time: 46h 15m 25s. Estimated total time: 53h 23m 48s. Time estimates for 10 more iterations: 10m 40s, 100 more iterations: 1h 46m 47s, 500 more iterations: 8h 53m 58s.
[2025-11-13 05:18:05,003][__main__][INFO] - Starting iteration 465.
[2025-11-13 05:18:05,489][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:18:05,490][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:18:24,905][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:18:24,947][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:18:28,907][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:18:35,378][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:18:36,880][__main__][INFO] - Number of regex retries in iteration 465: 4
[2025-11-13 05:18:36,881][__main__][INFO] - agents played in iteration 465 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:18:37,694][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:18:37,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:18:37,745][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:18:37,769][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:18:37,769][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:18:37,770][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:18:38,595][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:18:39,056][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:18:39,562][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:18:40,061][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:18:40,564][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:18:41,070][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:18:41,581][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:18:42,077][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:18:42,575][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:18:43,073][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:18:43,572][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:18:44,081][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:18:44,580][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:18:45,082][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:18:45,594][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:18:46,092][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:18:46,592][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:18:47,094][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:18:47,595][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:18:48,100][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:18:48,601][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:18:49,096][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:18:49,596][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:18:50,092][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:18:50,594][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:18:51,093][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:18:51,592][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:18:52,093][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:18:52,590][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:18:53,086][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:18:53,585][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:18:54,082][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:18:54,578][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:18:55,075][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:18:55,572][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:18:56,087][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:18:56,587][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:18:57,088][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:18:57,599][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:18:58,100][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:18:58,597][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:18:59,098][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:18:59,601][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:19:00,110][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:19:00,612][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:19:01,113][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:19:01,616][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:19:02,140][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:19:02,644][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:19:03,146][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:19:03,648][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:19:04,154][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:19:04,657][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:19:05,158][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:19:05,676][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:19:06,176][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:19:06,689][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:19:07,195][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:19:07,703][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:19:08,214][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:19:08,716][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:19:09,223][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:19:09,727][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:19:10,229][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:19:10,734][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10090 tokens.
[2025-11-13 05:19:11,562][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 05:19:12,321][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:19:12,323][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:19:12,325][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:19:13,422][__main__][INFO] - Iteration 466 took 1m 7s (46.21% Gen, 52.18% Train). Generation: 31s, Training: 35s. Estimated remaining time: 49h 27m 8s. Estimated total time: 56h 36m 40s. Time estimates for 10 more iterations: 11m 19s, 100 more iterations: 1h 53m 13s, 500 more iterations: 9h 26m 6s.
[2025-11-13 05:19:13,424][__main__][INFO] - Starting iteration 466.
[2025-11-13 05:19:13,913][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:19:13,913][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:19:28,620][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:19:37,773][__main__][INFO] - Number of regex retries in iteration 466: 1
[2025-11-13 05:19:37,773][__main__][INFO] - agents played in iteration 466 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:19:38,627][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:19:38,653][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:19:38,679][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:19:38,702][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:19:38,702][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:19:38,703][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:19:39,552][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:19:40,008][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:19:40,513][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:19:41,010][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:19:41,507][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:19:42,015][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:19:42,512][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:19:43,010][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:19:43,514][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:19:44,017][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:19:44,522][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:19:45,023][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:19:45,526][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:19:46,033][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:19:46,533][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:19:47,032][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:19:47,539][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:19:48,036][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:19:48,534][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:19:49,033][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:19:49,533][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:19:50,032][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:19:50,529][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:19:51,030][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:19:51,529][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:19:52,027][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:19:52,529][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:19:53,031][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:19:53,530][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:19:54,026][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:19:54,521][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:19:55,019][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:19:55,515][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:19:56,012][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:19:56,510][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:19:57,007][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:19:57,506][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:19:58,002][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:19:58,498][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:19:58,994][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:19:59,494][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:19:59,996][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:20:00,498][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:20:00,997][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:20:01,496][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:20:01,993][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:20:02,493][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:20:03,353][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:20:05,269][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:20:05,774][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:20:06,276][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:20:06,776][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:20:07,280][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:20:07,781][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:20:08,281][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:20:08,779][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:20:09,283][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:20:09,792][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:20:10,292][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:20:10,800][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:20:11,299][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:20:11,798][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:20:12,304][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:20:12,809][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:20:13,313][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10057 tokens.
[2025-11-13 05:20:14,207][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.21%, Current % of VRAM taken: 60.47%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:34
[2025-11-13 05:20:14,879][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:20:14,882][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:20:14,884][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:20:15,916][__main__][INFO] - Iteration 467 took 1m 2s (38.48% Gen, 59.85% Train). Generation: 23s, Training: 37s. Estimated remaining time: 44h 29m 36s. Estimated total time: 51h 40m 10s. Time estimates for 10 more iterations: 10m 20s, 100 more iterations: 1h 43m 20s, 500 more iterations: 8h 36m 41s.
[2025-11-13 05:20:15,920][__main__][INFO] - Starting iteration 467.
[2025-11-13 05:20:16,421][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:20:16,422][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:20:47,637][__main__][INFO] - Number of regex retries in iteration 467: 0
[2025-11-13 05:20:47,638][__main__][INFO] - agents played in iteration 467 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:20:48,418][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:20:48,445][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:20:48,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:20:48,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:20:48,496][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:20:48,497][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:20:49,285][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:20:49,744][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:20:50,251][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:20:50,755][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:20:51,254][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:20:51,755][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:20:52,257][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:20:52,759][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:20:53,261][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:20:53,758][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:20:54,257][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:20:54,759][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:20:55,260][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:20:55,762][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:20:56,261][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:20:56,761][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:20:57,260][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:20:57,761][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:20:58,270][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:20:58,771][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:20:59,269][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:20:59,766][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:21:00,262][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:21:00,763][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:21:01,258][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:21:01,755][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:21:02,254][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:21:02,752][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:21:03,252][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:21:03,756][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:21:04,253][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:21:04,752][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:21:05,249][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:21:05,747][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:21:06,243][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:21:06,739][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:21:07,241][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:21:07,737][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:21:08,235][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:21:08,734][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:21:09,231][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:21:09,731][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:21:10,259][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:21:10,766][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:21:11,271][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:21:11,774][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:21:12,275][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:21:12,776][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:21:13,278][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:21:13,779][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:21:14,285][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:21:14,784][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:21:15,287][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:21:15,795][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:21:16,295][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:21:16,806][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:21:17,310][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:21:17,831][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:21:18,331][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:21:18,832][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:21:19,340][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:21:19,839][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:21:20,342][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:21:20,844][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:21:21,344][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10095 tokens.
[2025-11-13 05:21:22,170][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 05:21:22,943][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:21:22,945][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:21:22,947][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:21:23,973][__main__][INFO] - Iteration 468 took 1m 7s (46.21% Gen, 52.27% Train). Generation: 31s, Training: 35s. Estimated remaining time: 49h 5m 56s. Estimated total time: 56h 17m 38s. Time estimates for 10 more iterations: 11m 15s, 100 more iterations: 1h 52m 35s, 500 more iterations: 9h 22m 56s.
[2025-11-13 05:21:23,975][__main__][INFO] - Starting iteration 468.
[2025-11-13 05:21:24,456][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:21:24,456][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:21:53,217][__main__][INFO] - Number of regex retries in iteration 468: 0
[2025-11-13 05:21:53,219][__main__][INFO] - agents played in iteration 468 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:21:54,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:21:54,094][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:21:54,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:21:54,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:21:54,142][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:21:54,142][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:21:54,974][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:21:55,431][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:21:56,068][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:21:56,574][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:21:57,075][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:21:57,578][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:21:58,076][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:21:58,578][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:21:59,085][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:21:59,582][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:22:00,079][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:22:00,576][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:22:01,076][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:22:01,577][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:22:02,074][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:22:02,572][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:22:03,069][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:22:03,567][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:22:04,062][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:22:04,558][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:22:05,057][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:22:05,554][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:22:06,051][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:22:06,553][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:22:07,051][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:22:07,553][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:22:08,052][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:22:08,556][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:22:09,058][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:22:09,559][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:22:10,062][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:22:10,562][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:22:11,062][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:22:11,559][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:22:12,057][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:22:12,556][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:22:13,057][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:22:13,557][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:22:14,058][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:22:14,558][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:22:15,058][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:22:15,566][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:22:16,065][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:22:16,565][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:22:17,068][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:22:17,567][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:22:18,065][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:22:18,568][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:22:19,067][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:22:19,575][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:22:20,082][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:22:20,592][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:22:21,106][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:22:21,608][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:22:22,117][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:22:22,618][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:22:23,122][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:22:23,639][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:22:24,142][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:22:24,644][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:22:25,150][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:22:25,653][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:22:26,159][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:22:26,661][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:22:27,164][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10119 tokens.
[2025-11-13 05:22:28,004][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:33
[2025-11-13 05:22:28,698][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:22:28,699][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:22:28,701][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:22:29,676][__main__][INFO] - Iteration 469 took 1m 5s (44.10% Gen, 54.40% Train). Generation: 28s, Training: 35s. Estimated remaining time: 47h 8m 14s. Estimated total time: 54h 21m 2s. Time estimates for 10 more iterations: 10m 52s, 100 more iterations: 1h 48m 42s, 500 more iterations: 9h 3m 30s.
[2025-11-13 05:22:29,678][__main__][INFO] - Starting iteration 469.
[2025-11-13 05:22:30,154][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:22:30,155][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:22:57,192][__main__][INFO] - Number of regex retries in iteration 469: 0
[2025-11-13 05:22:57,193][__main__][INFO] - agents played in iteration 469 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:22:58,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:22:58,089][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:22:58,116][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:22:58,140][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:22:58,140][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:22:58,141][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:22:58,998][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:22:59,453][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:22:59,954][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:23:00,457][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:23:00,963][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:23:01,462][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:23:01,958][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:23:02,456][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:23:02,957][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:23:03,454][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:23:03,951][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:23:04,450][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:23:04,947][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:23:05,448][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:23:05,945][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:23:06,449][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:23:06,953][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:23:07,455][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:23:07,975][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:23:08,475][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:23:08,974][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:23:09,476][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:23:09,978][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:23:10,477][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:23:10,976][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:23:11,479][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:23:11,975][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:23:12,476][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:23:12,977][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:23:13,475][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:23:13,976][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:23:14,474][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:23:14,973][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:23:15,473][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:23:15,971][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:23:16,469][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:23:16,965][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:23:17,469][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:23:17,965][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:23:18,461][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:23:18,959][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:23:19,455][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:23:19,955][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:23:20,454][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:23:20,951][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:23:21,450][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:23:21,951][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:23:22,447][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:23:22,944][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:23:23,441][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:23:23,953][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:23:24,454][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:23:24,956][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:23:25,455][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:23:25,954][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:23:26,456][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:23:26,955][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:23:27,457][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:23:27,959][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:23:28,463][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:23:28,967][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:23:29,466][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:23:29,967][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:23:30,473][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:23:30,977][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10129 tokens.
[2025-11-13 05:23:31,839][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 05:23:32,593][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:23:32,595][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:23:32,597][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:23:33,742][__main__][INFO] - Iteration 470 took 1m 3s (42.52% Gen, 55.68% Train). Generation: 27s, Training: 35s. Estimated remaining time: 45h 45m 34s. Estimated total time: 52h 59m 26s. Time estimates for 10 more iterations: 10m 35s, 100 more iterations: 1h 45m 58s, 500 more iterations: 8h 49m 54s.
[2025-11-13 05:23:33,744][__main__][INFO] - Starting iteration 470.
[2025-11-13 05:23:34,291][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 05:23:34,291][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:24:02,894][__main__][INFO] - Number of regex retries in iteration 470: 0
[2025-11-13 05:24:02,896][__main__][INFO] - agents played in iteration 470 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:24:03,745][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:24:03,774][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:24:03,801][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:24:03,824][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:24:03,824][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:24:03,825][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:24:04,565][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:24:05,020][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:24:05,522][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:24:06,023][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:24:06,526][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:24:07,027][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:24:07,538][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:24:08,040][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:24:08,537][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:24:09,033][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:24:09,530][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:24:10,040][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:24:10,539][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:24:11,035][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:24:11,533][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:24:12,030][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:24:12,528][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:24:13,030][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:24:13,528][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:24:14,042][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:24:14,542][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:24:15,037][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:24:15,534][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:24:16,033][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:24:16,540][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:24:17,038][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:24:17,539][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:24:18,051][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:24:18,552][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:24:19,053][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:24:19,549][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:24:20,047][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:24:20,556][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:24:21,080][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:24:21,577][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:24:22,073][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:24:22,570][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:24:23,068][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:24:23,571][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:24:24,067][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:24:24,565][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:24:25,062][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:24:25,563][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:24:26,060][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:24:26,556][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:24:27,059][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:24:27,557][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:24:28,059][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:24:28,557][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:24:29,060][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:24:29,566][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:24:30,069][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:24:30,573][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:24:31,078][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:24:31,579][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:24:32,085][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:24:32,589][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:24:33,089][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:24:33,590][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:24:34,094][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:24:34,594][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:24:35,094][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:24:35,597][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:24:36,101][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:24:36,605][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10090 tokens.
[2025-11-13 05:24:37,507][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.01%, Current % of VRAM taken: 60.27%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 05:24:38,121][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:24:38,123][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:24:38,124][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:24:40,133][__main__][INFO] - Iteration 471 took 1m 5s (43.44% Gen, 53.50% Train). Generation: 28s, Training: 35s. Estimated remaining time: 47h 37m 10s. Estimated total time: 54h 52m 9s. Time estimates for 10 more iterations: 10m 58s, 100 more iterations: 1h 49m 44s, 500 more iterations: 9h 8m 41s.
[2025-11-13 05:24:40,136][__main__][INFO] - Starting iteration 471.
[2025-11-13 05:24:40,609][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:24:40,609][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:25:03,834][__main__][INFO] - Number of regex retries in iteration 471: 0
[2025-11-13 05:25:03,834][__main__][INFO] - agents played in iteration 471 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:25:04,674][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:25:04,701][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:25:04,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:25:04,749][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:25:04,749][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:25:04,751][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:25:05,498][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:25:05,952][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:25:06,467][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:25:06,970][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:25:07,468][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:25:07,966][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:25:08,464][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:25:08,969][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:25:09,469][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:25:09,971][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:25:10,473][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:25:10,976][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:25:11,478][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:25:11,978][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:25:12,481][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:25:12,984][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:25:13,481][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:25:13,983][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:25:14,479][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:25:14,976][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:25:15,475][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:25:15,976][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:25:16,473][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:25:16,976][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:25:17,473][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:25:17,974][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:25:18,475][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:25:18,996][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:25:19,498][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:25:19,998][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:25:20,498][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:25:21,002][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:25:21,500][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:25:22,005][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:25:22,508][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:25:23,010][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:25:23,511][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:25:24,013][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:25:24,511][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:25:25,014][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:25:25,512][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:25:26,013][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:25:26,510][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:25:27,010][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:25:27,510][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:25:28,009][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:25:28,511][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:25:29,011][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:25:29,511][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:25:30,015][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:25:30,516][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:25:31,022][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:25:31,527][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:25:32,033][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:25:32,534][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:25:33,037][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:25:33,538][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:25:34,054][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:25:34,555][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:25:35,059][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:25:35,562][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:25:36,066][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:25:36,574][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:25:37,077][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:25:37,582][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10238 tokens.
[2025-11-13 05:25:38,457][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 05:25:39,226][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:25:39,228][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:25:39,230][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:25:40,193][__main__][INFO] - Iteration 472 took 59s (38.98% Gen, 59.40% Train). Generation: 23s, Training: 35s. Estimated remaining time: 42h 23m 16s. Estimated total time: 49h 39m 15s. Time estimates for 10 more iterations: 9m 55s, 100 more iterations: 1h 39m 18s, 500 more iterations: 8h 16m 32s.
[2025-11-13 05:25:40,195][__main__][INFO] - Starting iteration 472.
[2025-11-13 05:25:40,669][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:25:40,670][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:25:57,893][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:26:06,184][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 20 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:26:10,068][__main__][INFO] - Number of regex retries in iteration 472: 2
[2025-11-13 05:26:10,068][__main__][INFO] - agents played in iteration 472 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:26:10,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:26:10,988][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:26:11,014][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:26:11,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:26:11,038][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:26:11,039][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:26:11,835][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:26:12,292][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:26:12,799][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:26:13,303][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:26:13,808][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:26:14,307][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:26:14,807][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:26:15,304][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:26:15,802][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:26:16,301][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:26:16,800][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:26:17,301][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:26:17,802][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:26:18,300][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:26:18,808][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:26:19,308][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:26:19,810][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:26:20,308][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:26:20,806][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:26:21,305][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:26:21,802][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:26:22,299][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:26:22,798][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:26:23,296][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:26:23,798][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:26:24,296][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:26:24,796][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:26:25,299][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:26:25,797][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:26:26,296][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:26:26,792][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:26:27,290][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:26:27,795][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:26:28,294][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:26:28,791][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:26:29,289][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:26:29,792][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:26:30,289][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:26:30,786][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:26:31,282][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:26:31,788][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:26:32,293][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:26:32,797][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:26:33,299][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:26:33,800][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:26:34,309][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:26:34,812][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:26:35,316][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:26:35,820][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:26:36,325][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:26:36,830][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:26:37,328][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:26:37,832][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:26:38,335][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:26:38,837][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:26:39,339][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:26:39,843][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:26:40,346][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:26:40,848][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:26:41,353][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:26:41,861][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:26:42,369][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:26:42,871][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:26:43,375][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:26:43,882][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10091 tokens.
[2025-11-13 05:26:44,771][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 05:26:45,438][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:26:45,440][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:26:45,442][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:26:46,410][__main__][INFO] - Iteration 473 took 1m 5s (44.72% Gen, 53.81% Train). Generation: 29s, Training: 35s. Estimated remaining time: 47h 29m 59s. Estimated total time: 54h 47m 4s. Time estimates for 10 more iterations: 10m 57s, 100 more iterations: 1h 49m 34s, 500 more iterations: 9h 7m 50s.
[2025-11-13 05:26:46,412][__main__][INFO] - Starting iteration 473.
[2025-11-13 05:26:46,913][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:26:46,914][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:27:09,923][__main__][INFO] - Number of regex retries in iteration 473: 0
[2025-11-13 05:27:09,923][__main__][INFO] - agents played in iteration 473 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:27:10,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:27:10,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:27:10,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:27:10,869][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:27:10,870][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:27:10,870][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:27:11,692][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:27:12,146][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:27:12,666][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:27:13,162][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:27:13,663][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:27:14,159][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:27:14,655][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:27:15,153][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:27:15,651][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:27:16,150][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:27:16,656][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:27:17,154][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:27:17,654][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:27:18,156][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:27:18,654][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:27:19,165][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:27:19,664][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:27:20,160][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:27:20,656][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:27:21,151][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:27:21,650][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:27:22,146][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:27:22,643][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:27:23,140][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:27:23,636][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:27:24,138][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:27:24,635][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:27:25,133][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:27:25,633][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:27:26,131][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:27:26,634][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:27:27,131][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:27:27,629][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:27:28,130][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:27:28,628][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:27:29,125][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:27:29,625][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:27:30,126][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:27:30,626][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:27:31,122][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:27:31,620][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:27:32,121][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:27:32,616][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:27:33,112][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:27:33,606][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:27:34,106][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:27:34,599][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:27:35,095][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:27:35,594][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:27:36,094][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:27:36,596][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:27:37,098][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:27:37,609][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:27:38,112][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:27:38,616][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:27:39,117][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:27:39,618][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:27:40,116][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:27:40,620][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:27:41,126][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:27:41,628][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:27:42,130][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:27:42,634][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:27:43,135][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:27:43,637][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10045 tokens.
[2025-11-13 05:27:44,550][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 05:27:45,292][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:27:45,295][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:27:45,297][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:27:46,336][__main__][INFO] - Iteration 474 took 59s (38.72% Gen, 59.53% Train). Generation: 23s, Training: 35s. Estimated remaining time: 42h 13m 5s. Estimated total time: 49h 31m 9s. Time estimates for 10 more iterations: 9m 54s, 100 more iterations: 1h 39m 2s, 500 more iterations: 8h 15m 11s.
[2025-11-13 05:27:46,337][__main__][INFO] - Starting iteration 474.
[2025-11-13 05:27:46,909][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:27:46,910][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:28:16,619][__main__][INFO] - Number of regex retries in iteration 474: 0
[2025-11-13 05:28:16,621][__main__][INFO] - agents played in iteration 474 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:28:17,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:28:17,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:28:17,513][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:28:17,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:28:17,540][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:28:17,541][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:28:18,284][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:28:18,737][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:28:19,241][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:28:19,738][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:28:20,234][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:28:20,738][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:28:21,236][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:28:21,733][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:28:22,231][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:28:22,734][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:28:23,229][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:28:23,725][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:28:24,227][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:28:24,724][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:28:25,225][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:28:25,722][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:28:26,218][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:28:26,713][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:28:27,213][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:28:27,707][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:28:28,202][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:28:28,699][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:28:29,194][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:28:29,689][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:28:30,184][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:28:30,680][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:28:31,182][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:28:31,679][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:28:32,179][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:28:32,689][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:28:33,209][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:28:33,711][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:28:34,214][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:28:34,711][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:28:35,218][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:28:35,722][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:28:36,222][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:28:36,720][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:28:37,222][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:28:37,723][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:28:38,222][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:28:38,724][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:28:39,223][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:28:39,720][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:28:40,219][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:28:40,719][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:28:41,217][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:28:41,715][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:28:42,214][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:28:42,714][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:28:43,226][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:28:43,727][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:28:44,230][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:28:44,733][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:28:45,235][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:28:45,741][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:28:46,244][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:28:46,750][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:28:47,255][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:28:47,759][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:28:48,265][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:28:48,771][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:28:49,273][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:28:49,795][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:28:50,297][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10119 tokens.
[2025-11-13 05:28:51,211][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 05:28:51,835][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:28:51,837][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:28:51,839][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:28:52,708][__main__][INFO] - Iteration 475 took 1m 5s (45.15% Gen, 53.52% Train). Generation: 29s, Training: 35s. Estimated remaining time: 47h 30m 47s. Estimated total time: 54h 49m 58s. Time estimates for 10 more iterations: 10m 57s, 100 more iterations: 1h 49m 39s, 500 more iterations: 9h 8m 19s.
[2025-11-13 05:28:52,711][__main__][INFO] - Starting iteration 475.
[2025-11-13 05:28:53,196][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:28:53,196][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:29:12,361][__main__][INFO] - Number of regex retries in iteration 475: 0
[2025-11-13 05:29:12,363][__main__][INFO] - agents played in iteration 475 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:29:13,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:29:13,211][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:29:13,245][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:29:13,271][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:29:13,272][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:29:13,273][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:29:14,129][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:29:14,589][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:29:15,096][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:29:15,598][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:29:16,114][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:29:16,617][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:29:17,117][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:29:17,615][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:29:18,112][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:29:18,621][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:29:19,118][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:29:19,621][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:29:20,134][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:29:20,633][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:29:21,132][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:29:21,629][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:29:22,126][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:29:22,635][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:29:23,133][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:29:23,635][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:29:24,145][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:29:24,647][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:29:25,152][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:29:25,653][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:29:26,156][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:29:26,657][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:29:27,157][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:29:27,656][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:29:28,155][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:29:28,655][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:29:29,172][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:29:29,684][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:29:30,184][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:29:30,685][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:29:31,183][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:29:31,680][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:29:32,178][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:29:32,675][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:29:33,175][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:29:33,672][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:29:34,169][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:29:34,667][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:29:35,175][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:29:35,673][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:29:36,174][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:29:36,673][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:29:37,176][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:29:37,675][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:29:38,174][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:29:38,672][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:29:39,171][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:29:39,668][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:29:40,166][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:29:40,668][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:29:41,167][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:29:41,664][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:29:42,167][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:29:42,664][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:29:43,161][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:29:43,659][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:29:44,156][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:29:44,653][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:29:45,153][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:29:45,650][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:29:46,148][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10138 tokens.
[2025-11-13 05:29:46,915][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 05:29:47,655][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:29:47,657][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:29:47,658][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:29:48,706][__main__][INFO] - Iteration 476 took 55s (34.53% Gen, 63.58% Train). Generation: 19s, Training: 35s. Estimated remaining time: 38h 55m 27s. Estimated total time: 46h 15m 34s. Time estimates for 10 more iterations: 9m 15s, 100 more iterations: 1h 32m 31s, 500 more iterations: 7h 42m 35s.
[2025-11-13 05:29:48,709][__main__][INFO] - Starting iteration 476.
[2025-11-13 05:29:49,198][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:29:49,198][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:30:14,206][__main__][INFO] - Number of regex retries in iteration 476: 0
[2025-11-13 05:30:14,209][__main__][INFO] - agents played in iteration 476 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:30:15,112][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:30:15,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:30:15,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:30:15,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:30:15,205][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:30:15,205][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:30:16,071][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:30:16,605][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:30:17,117][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:30:17,618][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:30:18,123][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:30:18,625][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:30:19,126][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:30:19,625][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:30:20,130][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:30:20,639][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:30:21,139][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:30:21,646][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:30:22,150][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:30:22,649][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:30:23,149][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:30:23,645][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:30:24,141][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:30:24,641][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:30:25,139][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:30:25,639][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:30:26,136][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:30:26,635][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:30:27,137][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:30:27,634][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:30:28,135][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:30:28,641][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:30:29,139][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:30:29,637][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:30:30,133][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:30:30,631][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:30:31,136][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:30:31,634][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:30:32,135][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:30:32,645][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:30:33,145][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:30:33,644][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:30:34,144][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:30:34,639][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:30:35,141][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:30:35,637][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:30:36,134][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:30:36,633][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:30:37,130][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:30:37,630][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:30:38,131][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:30:38,631][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:30:39,140][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:30:39,639][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:30:40,136][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:30:40,632][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:30:41,129][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:30:41,634][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:30:42,130][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:30:42,625][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:30:43,126][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:30:43,626][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:30:44,121][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:30:44,620][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:30:45,120][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:30:45,625][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:30:46,124][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:30:46,620][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:30:47,132][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:30:47,632][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:30:48,131][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10216 tokens.
[2025-11-13 05:30:48,965][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 05:30:49,636][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:30:49,638][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:30:49,640][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:30:50,549][__main__][INFO] - Iteration 477 took 1m 1s (40.77% Gen, 57.75% Train). Generation: 25s, Training: 35s. Estimated remaining time: 43h 46m 27s. Estimated total time: 51h 7m 36s. Time estimates for 10 more iterations: 10m 13s, 100 more iterations: 1h 42m 15s, 500 more iterations: 8h 31m 16s.
[2025-11-13 05:30:50,552][__main__][INFO] - Starting iteration 477.
[2025-11-13 05:30:51,074][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:30:51,074][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:31:04,681][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:31:04,899][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:31:16,038][__main__][INFO] - Number of regex retries in iteration 477: 2
[2025-11-13 05:31:16,039][__main__][INFO] - agents played in iteration 477 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:31:16,868][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:31:16,896][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:31:16,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:31:16,945][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:31:16,945][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:31:16,946][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:31:17,735][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:31:18,188][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:31:18,706][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:31:19,205][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:31:19,703][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:31:20,204][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:31:20,700][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:31:21,204][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:31:21,703][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:31:22,201][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:31:22,699][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:31:23,201][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:31:23,702][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:31:24,198][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:31:24,698][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:31:25,197][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:31:25,695][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:31:26,194][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:31:26,696][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:31:27,193][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:31:27,690][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:31:28,188][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:31:28,685][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:31:29,183][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:31:29,679][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:31:30,180][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:31:30,675][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:31:31,172][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:31:31,670][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:31:32,167][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:31:32,664][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:31:33,160][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:31:33,655][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:31:34,153][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:31:34,650][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:31:35,148][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:31:35,662][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:31:36,163][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:31:36,663][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:31:37,164][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:31:37,664][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:31:38,169][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:31:38,667][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:31:39,166][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:31:39,668][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:31:40,172][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:31:40,674][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:31:41,184][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:31:41,683][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:31:42,180][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:31:42,678][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:31:43,176][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:31:43,675][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:31:44,172][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:31:44,672][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:31:45,169][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:31:45,669][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:31:46,170][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:31:46,666][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:31:47,169][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:31:47,672][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:31:48,172][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:31:48,673][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:31:49,172][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:31:49,671][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10141 tokens.
[2025-11-13 05:31:50,445][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 05:31:51,186][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:31:51,187][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:31:51,189][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:31:52,173][__main__][INFO] - Iteration 478 took 1m 1s (40.86% Gen, 57.53% Train). Generation: 24s, Training: 35s. Estimated remaining time: 43h 32m 49s. Estimated total time: 50h 55m 0s. Time estimates for 10 more iterations: 10m 11s, 100 more iterations: 1h 41m 50s, 500 more iterations: 8h 29m 10s.
[2025-11-13 05:31:52,175][__main__][INFO] - Starting iteration 478.
[2025-11-13 05:31:52,657][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:31:52,657][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:32:15,715][__main__][INFO] - Number of regex retries in iteration 478: 0
[2025-11-13 05:32:15,717][__main__][INFO] - agents played in iteration 478 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:32:16,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:32:16,721][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:32:16,758][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:32:16,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:32:16,790][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:32:16,791][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:32:17,652][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:32:18,199][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:32:18,712][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:32:19,222][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:32:19,719][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:32:20,236][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:32:20,740][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:32:21,242][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:32:21,761][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:32:22,259][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:32:22,758][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:32:23,263][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:32:23,761][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:32:24,269][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:32:24,768][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:32:25,265][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:32:25,773][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:32:26,273][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:32:26,777][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:32:27,274][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:32:27,773][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:32:28,273][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:32:28,776][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:32:29,277][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:32:29,778][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:32:30,275][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:32:30,773][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:32:31,269][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:32:31,767][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:32:32,268][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:32:32,765][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:32:33,261][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:32:33,768][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:32:34,264][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:32:34,765][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:32:35,261][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:32:35,761][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:32:36,268][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:32:36,768][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:32:37,266][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:32:37,768][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:32:38,264][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:32:38,766][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:32:39,260][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:32:39,764][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:32:40,271][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:32:40,770][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:32:41,271][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:32:41,771][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:32:42,271][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:32:42,774][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:32:43,274][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:32:43,770][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:32:44,272][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:32:44,769][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:32:45,271][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:32:45,771][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:32:46,271][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:32:46,795][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:32:47,296][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:32:47,797][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:32:48,302][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:32:48,802][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:32:49,301][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:32:49,801][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10156 tokens.
[2025-11-13 05:32:50,625][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:32
[2025-11-13 05:32:51,341][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:32:51,343][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:32:51,345][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:32:52,282][__main__][INFO] - Iteration 479 took 59s (38.67% Gen, 59.75% Train). Generation: 23s, Training: 35s. Estimated remaining time: 42h 18m 8s. Estimated total time: 49h 41m 19s. Time estimates for 10 more iterations: 9m 56s, 100 more iterations: 1h 39m 22s, 500 more iterations: 8h 16m 53s.
[2025-11-13 05:32:52,288][__main__][INFO] - Starting iteration 479.
[2025-11-13 05:32:52,777][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:32:52,778][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:33:20,757][__main__][INFO] - Number of regex retries in iteration 479: 0
[2025-11-13 05:33:20,757][__main__][INFO] - agents played in iteration 479 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:33:21,621][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:33:21,647][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:33:21,673][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:33:21,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:33:21,697][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:33:21,699][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:33:22,532][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:33:22,990][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:33:23,497][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:33:23,998][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:33:24,502][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:33:25,001][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:33:25,509][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:33:26,012][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:33:26,515][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:33:27,014][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:33:27,513][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:33:28,018][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:33:28,518][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:33:29,019][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:33:29,532][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:33:30,032][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:33:30,537][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:33:31,034][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:33:31,532][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:33:32,032][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:33:32,533][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:33:33,033][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:33:33,530][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:33:34,027][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:33:34,534][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:33:35,037][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:33:35,534][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:33:36,034][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:33:36,534][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:33:37,034][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:33:37,534][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:33:38,033][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:33:38,545][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:33:39,042][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:33:39,538][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:33:40,059][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:33:40,555][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:33:41,055][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:33:41,551][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:33:42,047][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:33:42,557][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:33:43,057][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:33:43,554][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:33:44,056][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:33:44,554][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:33:45,067][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:33:45,570][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:33:46,070][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:33:46,584][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:33:47,077][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:33:47,576][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:33:48,077][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:33:48,573][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:33:49,069][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:33:49,570][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:33:50,066][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:33:50,563][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:33:51,058][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:33:51,557][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:33:52,075][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:33:52,571][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:33:53,068][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:33:53,568][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:33:54,071][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:33:54,580][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10137 tokens.
[2025-11-13 05:33:55,440][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 05:33:56,237][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:33:56,238][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:33:56,240][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:33:57,295][__main__][INFO] - Iteration 480 took 1m 4s (43.37% Gen, 55.00% Train). Generation: 27s, Training: 35s. Estimated remaining time: 46h 21m 38s. Estimated total time: 53h 45m 53s. Time estimates for 10 more iterations: 10m 45s, 100 more iterations: 1h 47m 31s, 500 more iterations: 8h 57m 38s.
[2025-11-13 05:33:57,297][__main__][INFO] - Starting iteration 480.
[2025-11-13 05:33:57,778][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 05:33:57,779][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:34:20,111][__main__][INFO] - Number of regex retries in iteration 480: 0
[2025-11-13 05:34:20,113][__main__][INFO] - agents played in iteration 480 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:34:20,998][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:34:21,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:34:21,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:34:21,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:34:21,094][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:34:21,096][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:34:21,939][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:34:22,452][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:34:22,967][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:34:23,477][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:34:23,979][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:34:24,480][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:34:24,979][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:34:25,479][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:34:25,992][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:34:26,492][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:34:26,990][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:34:27,488][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:34:27,987][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:34:28,490][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:34:28,986][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:34:29,487][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:34:29,993][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:34:30,494][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:34:30,996][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:34:31,491][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:34:31,989][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:34:32,493][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:34:32,990][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:34:33,486][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:34:33,985][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:34:34,484][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:34:34,989][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:34:35,489][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:34:35,990][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:34:36,499][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:34:37,003][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:34:37,506][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:34:38,015][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:34:38,513][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:34:39,014][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:34:39,517][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:34:40,020][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:34:40,529][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:34:41,035][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:34:41,535][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:34:42,038][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:34:42,536][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:34:43,039][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:34:43,538][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:34:44,035][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:34:44,541][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:34:45,041][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:34:45,539][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:34:46,037][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:34:46,538][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:34:47,037][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:34:47,538][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:34:48,039][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:34:48,536][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:34:49,038][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:34:49,536][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:34:50,035][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:34:50,531][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:34:51,029][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:34:51,528][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:34:52,031][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:34:52,529][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:34:53,028][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:34:53,527][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:34:54,021][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10099 tokens.
[2025-11-13 05:34:54,850][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 05:34:55,504][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:34:55,505][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:34:55,507][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:34:57,542][__main__][INFO] - Iteration 481 took 59s (37.37% Gen, 59.22% Train). Generation: 22s, Training: 35s. Estimated remaining time: 42h 22m 57s. Estimated total time: 49h 48m 12s. Time estimates for 10 more iterations: 9m 57s, 100 more iterations: 1h 39m 36s, 500 more iterations: 8h 18m 2s.
[2025-11-13 05:34:57,544][__main__][INFO] - Starting iteration 481.
[2025-11-13 05:34:58,033][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:34:58,034][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:35:21,962][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:35:29,098][__main__][INFO] - Number of regex retries in iteration 481: 1
[2025-11-13 05:35:29,099][__main__][INFO] - agents played in iteration 481 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:35:29,937][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:35:29,966][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:35:29,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:35:30,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:35:30,023][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:35:30,025][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:35:30,794][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:35:31,251][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:35:31,756][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:35:32,254][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:35:32,756][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:35:33,254][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:35:33,756][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:35:34,262][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:35:34,763][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:35:35,264][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:35:35,762][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:35:36,257][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:35:36,752][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:35:37,249][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:35:37,745][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:35:38,243][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:35:38,739][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:35:39,239][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:35:39,739][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:35:40,237][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:35:40,743][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:35:41,243][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:35:41,746][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:35:42,251][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:35:42,749][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:35:43,248][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:35:43,746][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:35:44,244][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:35:44,741][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:35:45,238][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:35:45,736][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:35:46,238][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:35:46,739][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:35:47,236][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:35:47,732][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:35:48,230][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:35:48,728][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:35:49,225][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:35:49,721][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:35:50,227][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:35:50,723][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:35:51,226][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:35:51,723][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:35:52,223][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:35:52,729][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:35:53,228][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:35:53,729][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:35:54,233][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:35:54,733][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:35:55,232][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:35:55,734][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:35:56,236][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:35:56,736][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:35:57,238][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:35:57,738][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:35:58,242][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:35:58,743][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:35:59,249][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:35:59,750][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:36:00,251][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:36:00,751][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:36:01,251][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:36:01,748][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:36:02,254][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:36:02,757][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10123 tokens.
[2025-11-13 05:36:03,638][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 05:36:04,387][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:36:04,388][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:36:04,390][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:36:05,367][__main__][INFO] - Iteration 482 took 1m 7s (46.13% Gen, 52.41% Train). Generation: 31s, Training: 35s. Estimated remaining time: 48h 40m 21s. Estimated total time: 56h 6m 45s. Time estimates for 10 more iterations: 11m 13s, 100 more iterations: 1h 52m 13s, 500 more iterations: 9h 21m 7s.
[2025-11-13 05:36:05,369][__main__][INFO] - Starting iteration 482.
[2025-11-13 05:36:05,862][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:36:05,863][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:36:33,489][__main__][INFO] - Number of regex retries in iteration 482: 0
[2025-11-13 05:36:33,491][__main__][INFO] - agents played in iteration 482 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:36:34,324][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:36:34,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:36:34,369][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:36:34,392][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:36:34,392][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:36:34,393][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:36:35,174][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:36:35,630][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:36:36,135][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:36:36,639][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:36:37,138][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:36:37,637][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:36:38,142][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:36:38,641][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:36:39,143][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:36:39,639][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:36:40,140][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:36:40,637][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:36:41,135][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:36:41,636][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:36:42,138][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:36:42,636][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:36:43,137][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:36:43,638][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:36:44,139][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:36:44,638][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:36:45,138][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:36:45,645][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:36:46,144][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:36:46,641][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:36:47,146][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:36:47,645][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:36:48,146][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:36:48,646][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:36:49,143][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:36:49,641][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:36:50,139][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:36:50,637][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:36:51,137][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:36:51,634][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:36:52,136][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:36:52,634][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:36:53,135][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:36:53,638][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:36:54,136][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:36:54,639][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:36:55,135][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:36:55,636][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:36:56,136][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:36:56,635][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:36:57,134][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:36:57,637][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:36:58,138][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:36:58,639][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:36:59,140][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:36:59,638][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:37:00,139][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:37:00,639][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:37:01,139][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:37:01,637][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:37:02,138][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:37:02,637][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:37:03,135][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:37:03,634][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:37:04,141][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:37:04,643][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:37:05,145][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:37:05,647][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:37:06,148][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:37:06,648][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:37:07,146][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10081 tokens.
[2025-11-13 05:37:08,067][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 05:37:08,703][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:37:08,706][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:37:08,707][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:37:09,657][__main__][INFO] - Iteration 483 took 1m 3s (43.31% Gen, 55.20% Train). Generation: 27s, Training: 35s. Estimated remaining time: 45h 42m 18s. Estimated total time: 53h 9m 46s. Time estimates for 10 more iterations: 10m 37s, 100 more iterations: 1h 46m 19s, 500 more iterations: 8h 51m 37s.
[2025-11-13 05:37:09,659][__main__][INFO] - Starting iteration 483.
[2025-11-13 05:37:10,144][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:37:10,144][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:37:35,186][__main__][INFO] - Number of regex retries in iteration 483: 0
[2025-11-13 05:37:35,187][__main__][INFO] - agents played in iteration 483 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:37:36,327][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:37:36,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:37:36,377][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:37:36,400][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:37:36,401][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:37:36,402][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:37:37,251][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:37:37,709][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:37:38,225][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:37:38,733][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:37:39,236][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:37:39,739][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:37:40,245][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:37:40,753][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:37:41,253][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:37:41,751][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:37:42,253][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:37:42,752][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:37:43,263][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:37:43,761][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:37:44,261][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:37:44,762][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:37:45,261][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:37:45,760][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:37:46,260][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:37:46,762][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:37:47,265][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:37:47,763][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:37:48,263][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:37:48,768][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:37:49,267][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:37:49,768][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:37:50,268][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:37:50,767][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:37:51,266][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:37:51,764][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:37:52,262][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:37:52,764][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:37:53,263][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:37:53,761][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:37:54,260][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:37:54,760][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:37:55,261][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:37:55,763][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:37:56,265][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:37:56,769][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:37:57,267][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:37:57,766][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:37:58,269][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:37:58,767][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:37:59,270][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:37:59,772][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:38:00,271][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:38:00,771][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:38:01,271][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:38:01,771][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:38:02,298][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:38:02,800][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:38:03,308][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:38:03,807][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:38:04,315][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:38:04,817][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:38:05,315][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:38:05,830][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:38:06,330][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:38:06,829][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:38:07,326][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:38:07,826][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:38:08,339][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:38:08,842][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:38:09,343][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10133 tokens.
[2025-11-13 05:38:10,262][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:33
[2025-11-13 05:38:11,056][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:38:11,058][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:38:11,059][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:38:12,049][__main__][INFO] - Iteration 484 took 1m 1s (40.45% Gen, 57.95% Train). Generation: 25s, Training: 35s. Estimated remaining time: 44h 6m 47s. Estimated total time: 51h 35m 17s. Time estimates for 10 more iterations: 10m 19s, 100 more iterations: 1h 43m 10s, 500 more iterations: 8h 35m 52s.
[2025-11-13 05:38:12,052][__main__][INFO] - Starting iteration 484.
[2025-11-13 05:38:12,616][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:38:12,616][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:38:46,152][__main__][INFO] - Number of regex retries in iteration 484: 0
[2025-11-13 05:38:46,154][__main__][INFO] - agents played in iteration 484 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:38:46,998][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:38:47,028][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:38:47,052][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:38:47,074][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:38:47,075][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:38:47,076][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:38:47,920][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:38:48,379][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:38:48,882][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:38:49,384][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:38:49,883][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:38:50,386][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:38:50,885][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:38:51,394][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:38:51,895][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:38:52,395][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:38:52,913][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:38:53,416][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:38:53,920][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:38:54,444][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:38:54,943][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:38:55,449][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:38:55,949][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:38:56,449][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:38:56,951][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:38:57,448][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:38:57,947][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:38:58,444][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:38:58,949][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:38:59,455][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:38:59,957][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:39:00,459][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:39:00,960][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:39:01,462][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:39:01,964][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:39:02,466][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:39:02,964][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:39:03,463][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:39:03,960][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:39:04,458][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:39:04,961][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:39:05,461][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:39:05,965][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:39:06,465][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:39:06,968][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:39:07,467][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:39:07,966][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:39:08,467][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:39:08,966][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:39:09,469][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:39:09,973][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:39:10,503][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:39:11,007][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:39:11,517][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:39:12,023][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:39:12,545][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:39:13,044][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:39:13,546][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:39:14,047][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:39:14,547][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:39:15,051][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:39:15,555][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:39:16,065][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:39:16,571][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:39:17,074][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:39:17,576][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:39:18,079][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:39:18,583][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:39:19,090][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:39:19,595][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:39:20,097][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10132 tokens.
[2025-11-13 05:39:20,981][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:33
[2025-11-13 05:39:21,614][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:39:21,616][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:39:21,618][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:39:22,524][__main__][INFO] - Iteration 485 took 1m 9s (47.97% Gen, 50.73% Train). Generation: 33s, Training: 35s. Estimated remaining time: 50h 45m 47s. Estimated total time: 58h 15m 28s. Time estimates for 10 more iterations: 11m 39s, 100 more iterations: 1h 56m 30s, 500 more iterations: 9h 42m 34s.
[2025-11-13 05:39:22,527][__main__][INFO] - Starting iteration 485.
[2025-11-13 05:39:23,008][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:39:23,009][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:39:38,637][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:39:49,326][__main__][INFO] - Number of regex retries in iteration 485: 1
[2025-11-13 05:39:49,327][__main__][INFO] - agents played in iteration 485 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:39:50,157][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:39:50,183][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:39:50,207][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:39:50,229][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:39:50,230][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:39:50,231][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:39:51,072][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:39:51,529][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:39:52,044][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:39:52,545][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:39:53,591][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:39:54,827][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:39:55,328][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:39:55,831][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:39:56,336][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:39:56,837][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:39:57,352][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:39:57,857][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:39:58,365][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:39:58,869][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:39:59,371][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:39:59,872][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:40:00,372][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:40:00,877][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:40:01,387][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:40:01,889][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:40:02,421][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:40:02,926][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:40:03,430][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:40:03,950][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:40:04,453][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:40:04,958][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:40:05,460][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:40:05,958][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:40:06,464][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:40:06,960][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:40:07,464][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:40:07,963][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:40:08,460][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:40:08,963][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:40:09,460][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:40:09,956][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:40:10,461][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:40:10,959][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:40:11,458][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:40:11,954][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:40:12,452][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:40:12,955][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:40:13,453][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:40:13,949][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:40:14,457][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:40:14,954][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:40:15,454][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:40:15,950][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:40:16,447][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:40:16,956][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:40:17,457][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:40:17,958][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:40:18,472][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:40:18,981][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:40:19,486][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:40:19,986][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:40:20,490][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:40:20,991][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:40:21,492][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:40:21,993][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:40:22,496][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:40:22,999][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:40:23,501][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:40:24,007][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:40:24,507][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10112 tokens.
[2025-11-13 05:40:25,401][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:34
[2025-11-13 05:40:26,036][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:40:26,037][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:40:26,040][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:40:26,989][__main__][INFO] - Iteration 486 took 1m 3s (41.13% Gen, 57.38% Train). Generation: 26s, Training: 36s. Estimated remaining time: 45h 48m 21s. Estimated total time: 53h 19m 6s. Time estimates for 10 more iterations: 10m 39s, 100 more iterations: 1h 46m 38s, 500 more iterations: 8h 53m 11s.
[2025-11-13 05:40:26,992][__main__][INFO] - Starting iteration 486.
[2025-11-13 05:40:27,477][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:40:27,478][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:40:52,934][__main__][INFO] - Number of regex retries in iteration 486: 0
[2025-11-13 05:40:52,935][__main__][INFO] - agents played in iteration 486 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:40:53,725][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:40:53,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:40:53,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:40:53,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:40:53,803][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:40:53,805][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:40:54,620][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:40:55,089][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:40:55,600][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:40:56,102][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:40:56,605][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:40:57,104][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:40:57,604][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:40:58,106][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:40:58,606][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:40:59,109][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:40:59,610][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:41:00,117][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:41:00,623][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:41:01,128][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:41:01,631][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:41:02,141][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:41:02,641][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:41:03,144][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:41:03,672][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:41:04,193][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:41:04,696][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:41:05,200][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:41:05,705][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:41:06,211][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:41:06,716][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:41:07,218][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:41:07,722][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:41:08,227][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:41:08,728][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:41:09,228][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:41:09,727][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:41:10,230][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:41:10,731][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:41:11,235][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:41:11,735][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:41:12,245][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:41:12,747][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:41:13,254][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:41:13,752][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:41:14,254][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:41:14,756][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:41:15,259][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:41:15,757][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:41:16,260][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:41:16,761][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:41:17,275][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:41:17,773][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:41:18,272][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:41:18,776][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:41:19,274][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:41:19,776][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:41:20,276][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:41:20,779][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:41:21,285][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:41:21,783][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:41:22,279][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:41:22,780][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:41:23,278][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:41:23,778][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:41:24,277][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:41:24,778][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:41:25,292][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:41:25,799][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:41:26,299][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:41:26,803][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10198 tokens.
[2025-11-13 05:41:27,636][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.35%, ΔTime: 00:00:33
[2025-11-13 05:41:28,406][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:41:28,408][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:41:28,411][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:41:29,410][__main__][INFO] - Iteration 487 took 1m 1s (41.10% Gen, 57.28% Train). Generation: 25s, Training: 35s. Estimated remaining time: 44h 4m 52s. Estimated total time: 51h 36m 39s. Time estimates for 10 more iterations: 10m 19s, 100 more iterations: 1h 43m 13s, 500 more iterations: 8h 36m 6s.
[2025-11-13 05:41:29,412][__main__][INFO] - Starting iteration 487.
[2025-11-13 05:41:29,879][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:41:29,879][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:41:54,846][__main__][INFO] - Number of regex retries in iteration 487: 0
[2025-11-13 05:41:54,848][__main__][INFO] - agents played in iteration 487 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:41:55,730][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:41:55,759][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:41:55,787][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:41:55,824][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:41:55,825][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:41:55,826][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:41:56,662][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:41:57,130][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:41:57,638][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:41:58,145][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:41:58,650][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:41:59,155][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:41:59,659][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:42:00,163][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:42:00,669][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:42:01,171][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:42:01,671][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:42:02,174][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:42:02,685][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:42:03,184][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:42:03,688][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:42:04,190][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:42:04,690][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:42:05,199][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:42:05,699][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:42:06,201][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:42:06,705][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:42:07,204][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:42:07,706][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:42:08,210][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:42:08,714][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:42:09,215][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:42:09,715][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:42:10,214][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:42:10,721][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:42:11,223][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:42:11,722][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:42:12,221][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:42:12,724][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:42:13,224][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:42:13,725][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:42:14,226][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:42:14,730][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:42:15,231][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:42:15,734][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:42:16,243][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:42:16,744][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:42:17,252][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:42:17,753][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:42:18,258][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:42:18,761][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:42:19,262][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:42:19,779][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:42:20,278][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:42:20,775][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:42:21,277][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:42:21,774][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:42:22,276][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:42:22,778][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:42:23,276][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:42:23,775][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:42:24,272][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:42:24,769][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:42:25,266][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:42:25,765][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:42:26,265][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:42:26,765][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:42:27,264][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:42:27,773][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:42:28,274][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:42:28,773][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10124 tokens.
[2025-11-13 05:42:29,632][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.06%, ΔTime: 00:00:32
[2025-11-13 05:42:30,308][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:42:30,310][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:42:30,312][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:42:31,264][__main__][INFO] - Iteration 488 took 1m 1s (40.67% Gen, 57.77% Train). Generation: 24s, Training: 35s. Estimated remaining time: 43h 36m 27s. Estimated total time: 51h 9m 17s. Time estimates for 10 more iterations: 10m 13s, 100 more iterations: 1h 42m 18s, 500 more iterations: 8h 31m 32s.
[2025-11-13 05:42:31,266][__main__][INFO] - Starting iteration 488.
[2025-11-13 05:42:31,756][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:42:31,757][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:43:00,191][__main__][INFO] - Number of regex retries in iteration 488: 0
[2025-11-13 05:43:00,192][__main__][INFO] - agents played in iteration 488 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:43:01,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:43:01,033][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:43:01,055][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:43:01,077][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:43:01,078][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:43:01,079][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:43:01,890][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:43:02,345][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:43:02,851][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:43:03,347][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:43:03,845][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:43:04,346][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:43:04,849][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:43:05,353][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:43:05,855][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:43:06,361][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:43:06,867][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:43:07,372][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:43:07,873][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:43:08,381][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:43:08,879][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:43:09,380][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:43:09,885][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:43:10,386][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:43:10,888][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:43:11,388][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:43:11,889][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:43:12,390][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:43:12,898][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:43:13,397][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:43:13,902][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:43:14,405][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:43:14,924][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:43:15,425][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:43:15,926][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:43:16,428][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:43:16,929][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:43:17,439][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:43:17,942][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:43:18,448][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:43:18,952][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:43:19,454][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:43:19,958][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:43:20,460][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:43:20,961][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:43:21,464][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:43:21,970][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:43:22,468][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:43:22,969][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:43:23,471][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:43:23,971][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:43:24,473][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:43:24,976][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:43:25,480][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:43:25,977][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:43:26,476][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:43:26,999][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:43:27,506][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:43:28,021][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:43:28,521][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:43:29,021][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:43:29,524][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:43:30,024][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:43:30,525][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:43:31,027][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:43:31,530][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:43:32,044][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:43:32,546][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:43:33,047][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:43:33,550][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:43:34,052][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10101 tokens.
[2025-11-13 05:43:34,905][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:33
[2025-11-13 05:43:35,639][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:43:35,640][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:43:35,642][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:43:36,656][__main__][INFO] - Iteration 489 took 1m 4s (43.81% Gen, 54.62% Train). Generation: 28s, Training: 35s. Estimated remaining time: 46h 31m 6s. Estimated total time: 54h 5m 1s. Time estimates for 10 more iterations: 10m 49s, 100 more iterations: 1h 48m 10s, 500 more iterations: 9h 0m 50s.
[2025-11-13 05:43:36,658][__main__][INFO] - Starting iteration 489.
[2025-11-13 05:43:37,159][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:43:37,160][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:43:51,119][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:44:02,687][__main__][INFO] - Number of regex retries in iteration 489: 1
[2025-11-13 05:44:02,688][__main__][INFO] - agents played in iteration 489 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:44:03,526][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:44:03,558][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:44:03,587][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:44:03,611][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:44:03,611][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:44:03,613][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:44:04,377][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:44:04,832][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:44:05,338][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:44:05,840][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:44:06,350][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:44:06,849][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:44:07,348][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:44:07,857][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:44:08,360][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:44:08,867][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:44:09,367][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:44:09,863][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:44:10,361][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:44:10,859][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:44:11,356][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:44:11,853][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:44:12,353][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:44:12,850][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:44:13,349][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:44:13,848][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:44:14,346][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:44:14,844][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:44:15,348][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:44:15,845][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:44:16,345][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:44:16,852][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:44:17,353][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:44:17,855][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:44:18,361][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:44:18,869][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:44:19,372][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:44:19,872][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:44:20,376][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:44:20,879][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:44:21,381][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:44:21,886][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:44:22,387][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:44:22,892][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:44:23,397][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:44:23,899][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:44:24,402][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:44:24,914][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:44:25,414][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:44:25,922][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:44:26,422][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:44:26,926][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:44:27,442][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:44:27,941][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:44:28,444][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:44:28,947][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:44:29,448][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:44:29,955][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:44:30,458][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:44:30,959][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:44:31,462][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:44:31,962][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:44:32,469][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:44:32,970][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:44:33,474][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:44:33,978][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:44:34,482][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:44:34,983][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:44:35,486][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:44:35,988][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:44:36,494][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10155 tokens.
[2025-11-13 05:44:37,365][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 05:44:38,019][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:44:38,021][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:44:38,023][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:44:38,921][__main__][INFO] - Iteration 490 took 1m 1s (41.33% Gen, 57.21% Train). Generation: 25s, Training: 35s. Estimated remaining time: 43h 53m 11s. Estimated total time: 51h 28m 8s. Time estimates for 10 more iterations: 10m 17s, 100 more iterations: 1h 42m 56s, 500 more iterations: 8h 34m 41s.
[2025-11-13 05:44:38,924][__main__][INFO] - Starting iteration 490.
[2025-11-13 05:44:39,406][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 05:44:39,407][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:45:03,223][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:45:07,956][__main__][INFO] - Number of regex retries in iteration 490: 1
[2025-11-13 05:45:07,956][__main__][INFO] - agents played in iteration 490 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:45:08,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:45:08,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:45:08,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:45:08,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:45:08,803][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:45:08,804][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:45:09,634][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:45:10,095][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:45:10,602][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:45:11,101][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:45:11,598][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:45:12,094][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:45:12,592][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:45:13,092][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:45:13,589][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:45:14,086][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:45:14,584][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:45:15,085][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:45:15,586][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:45:16,085][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:45:16,581][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:45:17,078][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:45:17,578][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:45:18,076][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:45:18,574][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:45:19,071][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:45:19,568][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:45:20,072][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:45:20,574][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:45:21,072][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:45:21,570][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:45:22,073][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:45:22,577][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:45:23,077][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:45:23,575][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:45:24,073][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:45:24,573][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:45:25,072][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:45:25,572][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:45:26,070][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:45:26,577][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:45:27,073][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:45:27,571][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:45:28,076][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:45:28,574][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:45:29,074][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:45:29,574][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:45:30,077][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:45:30,581][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:45:31,083][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:45:31,583][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:45:32,079][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:45:32,583][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:45:33,087][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:45:33,590][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:45:34,094][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:45:34,597][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:45:35,099][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:45:35,599][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:45:36,102][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:45:36,605][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:45:37,111][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:45:37,614][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:45:38,113][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:45:38,620][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:45:39,118][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:45:39,620][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:45:40,134][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:45:40,635][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:45:41,149][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:45:41,654][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10154 tokens.
[2025-11-13 05:45:42,533][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-13 05:45:43,265][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:45:43,267][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:45:43,269][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:45:45,155][__main__][INFO] - Iteration 491 took 1m 5s (43.42% Gen, 53.71% Train). Generation: 28s, Training: 35s. Estimated remaining time: 47h 11m 25s. Estimated total time: 54h 47m 28s. Time estimates for 10 more iterations: 10m 57s, 100 more iterations: 1h 49m 34s, 500 more iterations: 9h 7m 54s.
[2025-11-13 05:45:45,158][__main__][INFO] - Starting iteration 491.
[2025-11-13 05:45:45,632][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:45:45,632][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:46:02,942][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:46:03,109][mllm.models.large_language_model_local][WARNING] - Response Proposals: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:46:15,564][__main__][INFO] - Number of regex retries in iteration 491: 2
[2025-11-13 05:46:15,565][__main__][INFO] - agents played in iteration 491 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:46:16,422][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:46:16,450][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:46:16,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:46:16,514][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:46:16,514][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:46:16,515][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:46:17,338][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:46:17,795][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:46:18,300][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:46:18,799][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:46:19,296][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:46:19,793][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:46:20,291][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:46:20,792][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:46:21,287][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:46:21,789][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:46:22,285][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:46:22,782][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:46:23,279][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:46:23,776][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:46:24,272][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:46:24,772][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:46:25,268][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:46:25,783][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:46:26,281][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:46:26,786][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:46:27,282][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:46:27,780][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:46:28,282][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:46:28,780][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:46:29,282][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:46:29,779][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:46:30,275][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:46:30,775][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:46:31,274][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:46:31,773][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:46:32,284][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:46:32,783][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:46:33,286][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:46:33,788][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:46:34,285][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:46:34,782][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:46:35,279][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:46:35,777][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:46:36,294][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:46:36,792][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:46:37,293][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:46:37,795][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:46:38,296][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:46:38,802][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:46:39,301][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:46:39,804][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:46:40,311][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:46:40,813][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:46:41,320][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:46:41,821][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:46:42,326][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:46:42,829][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:46:43,331][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:46:43,835][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:46:44,337][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:46:44,838][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:46:45,342][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:46:45,842][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:46:46,343][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:46:46,846][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:46:47,348][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:46:47,850][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:46:48,365][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:46:48,864][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:46:49,379][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10088 tokens.
[2025-11-13 05:46:50,259][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 62.08%, ΔTime: 00:00:32
[2025-11-13 05:46:50,908][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:46:50,911][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:46:50,913][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:46:51,837][__main__][INFO] - Iteration 492 took 1m 6s (45.21% Gen, 53.39% Train). Generation: 29s, Training: 35s. Estimated remaining time: 47h 33m 8s. Estimated total time: 55h 10m 18s. Time estimates for 10 more iterations: 11m 2s, 100 more iterations: 1h 50m 20s, 500 more iterations: 9h 11m 43s.
[2025-11-13 05:46:51,839][__main__][INFO] - Starting iteration 492.
[2025-11-13 05:46:52,328][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:46:52,329][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:47:18,487][__main__][INFO] - Number of regex retries in iteration 492: 0
[2025-11-13 05:47:18,488][__main__][INFO] - agents played in iteration 492 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:47:19,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:47:19,325][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:47:19,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:47:19,371][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:47:19,372][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:47:19,373][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:47:20,133][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:47:20,593][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:47:21,098][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:47:21,600][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:47:22,107][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:47:22,609][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:47:23,108][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:47:23,606][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:47:24,106][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:47:24,614][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:47:25,114][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:47:25,613][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:47:26,114][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:47:26,612][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:47:27,110][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:47:27,607][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:47:28,104][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:47:28,610][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:47:29,107][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:47:29,604][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:47:30,104][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:47:30,600][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:47:31,101][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:47:31,597][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:47:32,094][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:47:32,593][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:47:33,092][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:47:33,591][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:47:34,090][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:47:34,587][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:47:35,096][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:47:35,593][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:47:36,092][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:47:36,590][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:47:37,086][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:47:37,582][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:47:38,079][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:47:38,579][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:47:39,094][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:47:39,592][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:47:40,091][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:47:40,595][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:47:41,094][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:47:41,599][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:47:42,098][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:47:42,598][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:47:43,114][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:47:43,617][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:47:44,123][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:47:44,623][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:47:45,125][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:47:45,632][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:47:46,136][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:47:46,643][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:47:47,146][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:47:47,649][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:47:48,155][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:47:48,655][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:47:49,157][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:47:49,658][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:47:50,162][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:47:50,663][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:47:51,179][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:47:51,680][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:47:52,193][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10058 tokens.
[2025-11-13 05:47:53,064][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 05:47:53,822][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:47:53,824][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:47:53,826][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:47:54,805][__main__][INFO] - Iteration 493 took 1m 2s (41.87% Gen, 56.56% Train). Generation: 26s, Training: 35s. Estimated remaining time: 44h 25m 40s. Estimated total time: 52h 3m 53s. Time estimates for 10 more iterations: 10m 24s, 100 more iterations: 1h 44m 7s, 500 more iterations: 8h 40m 38s.
[2025-11-13 05:47:54,807][__main__][INFO] - Starting iteration 493.
[2025-11-13 05:47:55,280][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:47:55,281][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:48:23,477][__main__][INFO] - Number of regex retries in iteration 493: 0
[2025-11-13 05:48:23,480][__main__][INFO] - agents played in iteration 493 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:48:24,322][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:48:24,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:48:24,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:48:24,399][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:48:24,400][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:48:24,401][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:48:25,158][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:48:25,613][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:48:26,119][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:48:26,620][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:48:27,118][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:48:27,617][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:48:28,117][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:48:28,616][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:48:29,118][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:48:29,618][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:48:30,118][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:48:30,615][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:48:31,112][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:48:31,611][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:48:32,109][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:48:32,613][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:48:33,120][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:48:33,616][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:48:34,119][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:48:34,617][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:48:35,116][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:48:35,618][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:48:36,114][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:48:36,611][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:48:37,113][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:48:37,608][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:48:38,103][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:48:38,601][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:48:39,096][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:48:39,601][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:48:40,097][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:48:40,593][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:48:41,102][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:48:41,598][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:48:42,094][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:48:42,597][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:48:43,098][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:48:43,604][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:48:44,104][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:48:44,604][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:48:45,142][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:48:45,646][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:48:46,149][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:48:46,657][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:48:47,159][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:48:47,664][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:48:48,164][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:48:48,668][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:48:49,171][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:48:49,675][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:48:50,182][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:48:50,682][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:48:51,188][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:48:51,704][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:48:52,205][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:48:52,707][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:48:53,210][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:48:53,711][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:48:54,216][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:48:54,721][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:48:55,228][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:48:55,738][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:48:56,240][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:48:56,744][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:48:57,246][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10139 tokens.
[2025-11-13 05:48:58,135][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.11%, ΔTime: 00:00:32
[2025-11-13 05:48:58,777][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:48:58,778][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:48:58,781][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:48:59,739][__main__][INFO] - Iteration 494 took 1m 4s (43.74% Gen, 54.76% Train). Generation: 28s, Training: 35s. Estimated remaining time: 46h 3m 41s. Estimated total time: 53h 42m 59s. Time estimates for 10 more iterations: 10m 44s, 100 more iterations: 1h 47m 25s, 500 more iterations: 8h 57m 9s.
[2025-11-13 05:48:59,742][__main__][INFO] - Starting iteration 494.
[2025-11-13 05:49:00,215][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:49:00,215][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:49:29,549][__main__][INFO] - Number of regex retries in iteration 494: 0
[2025-11-13 05:49:29,550][__main__][INFO] - agents played in iteration 494 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:49:30,390][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:49:30,416][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:49:30,441][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:49:30,464][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:49:30,464][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:49:30,465][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:49:31,231][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:49:31,686][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:49:32,202][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:49:32,706][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:49:33,208][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:49:33,709][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:49:34,214][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:49:34,714][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:49:35,215][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:49:35,716][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:49:36,228][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:49:36,730][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:49:37,231][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:49:37,729][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:49:38,226][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:49:38,723][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:49:39,222][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:49:39,717][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:49:40,224][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:49:40,723][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:49:41,225][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:49:41,729][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:49:42,225][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:49:42,725][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:49:43,221][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:49:43,718][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:49:44,216][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:49:44,713][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:49:45,210][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:49:45,704][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:49:46,199][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:49:46,715][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:49:47,212][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:49:47,709][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:49:48,209][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:49:48,707][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:49:49,211][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:49:49,716][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:49:50,218][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:49:50,727][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:49:51,230][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:49:51,730][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:49:52,228][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:49:52,727][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:49:53,230][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:49:53,729][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:49:54,231][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:49:54,736][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:49:55,243][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:49:55,749][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:49:56,250][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:49:56,751][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:49:57,255][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:49:57,759][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:49:58,260][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:49:58,765][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:49:59,268][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:49:59,785][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:50:00,288][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:50:00,793][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:50:01,345][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:50:03,016][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:50:03,522][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:50:04,020][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:50:04,524][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10127 tokens.
[2025-11-13 05:50:05,444][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:34
[2025-11-13 05:50:06,110][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:50:06,111][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:50:06,113][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:50:07,133][__main__][INFO] - Iteration 495 took 1m 6s (43.84% Gen, 54.64% Train). Generation: 29s, Training: 36s. Estimated remaining time: 48h 5m 31s. Estimated total time: 55h 45m 57s. Time estimates for 10 more iterations: 11m 9s, 100 more iterations: 1h 51m 31s, 500 more iterations: 9h 17m 39s.
[2025-11-13 05:50:07,136][__main__][INFO] - Starting iteration 495.
[2025-11-13 05:50:07,630][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:50:07,631][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:50:37,919][__main__][INFO] - Number of regex retries in iteration 495: 0
[2025-11-13 05:50:37,920][__main__][INFO] - agents played in iteration 495 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:50:38,747][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:50:38,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:50:38,797][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:50:38,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:50:38,820][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:50:38,821][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:50:39,596][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:50:40,046][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:50:40,553][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:50:41,053][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:50:41,554][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:50:42,056][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:50:42,559][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:50:43,057][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:50:43,561][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:50:44,058][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:50:44,555][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:50:45,053][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:50:45,549][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:50:46,046][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:50:46,542][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:50:47,043][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:50:47,542][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:50:48,041][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:50:48,540][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:50:49,040][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:50:49,538][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:50:50,038][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:50:50,541][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:50:51,041][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:50:51,541][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:50:52,036][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:50:52,539][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:50:53,036][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:50:53,538][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:50:54,038][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:50:54,538][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:50:55,043][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:50:55,541][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:50:56,044][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:50:56,545][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:50:57,045][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:50:57,546][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:50:58,050][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:50:58,553][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:50:59,066][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:50:59,573][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:51:00,077][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:51:00,580][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:51:01,084][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:51:01,586][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:51:02,089][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:51:02,615][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:51:03,122][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:51:03,622][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:51:04,133][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:51:04,637][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:51:05,137][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:51:05,644][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:51:06,144][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:51:06,646][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:51:07,147][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:51:07,648][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:51:08,155][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:51:08,655][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:51:09,156][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:51:09,658][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:51:10,160][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:51:10,665][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:51:11,167][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:51:11,671][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10128 tokens.
[2025-11-13 05:51:12,560][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 05:51:13,361][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:51:13,362][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:51:13,364][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:51:14,382][__main__][INFO] - Iteration 496 took 1m 6s (45.37% Gen, 53.10% Train). Generation: 30s, Training: 35s. Estimated remaining time: 47h 56m 4s. Estimated total time: 55h 37m 36s. Time estimates for 10 more iterations: 11m 7s, 100 more iterations: 1h 51m 15s, 500 more iterations: 9h 16m 16s.
[2025-11-13 05:51:14,384][__main__][INFO] - Starting iteration 496.
[2025-11-13 05:51:14,872][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:51:14,872][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:51:38,964][__main__][INFO] - Number of regex retries in iteration 496: 0
[2025-11-13 05:51:38,964][__main__][INFO] - agents played in iteration 496 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:51:39,815][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:51:39,841][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:51:39,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:51:39,890][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:51:39,891][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:51:39,892][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:51:40,663][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:51:41,133][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:51:41,638][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:51:42,137][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:51:42,636][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:51:43,138][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:51:43,646][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:51:44,145][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:51:44,642][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:51:45,142][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:51:45,642][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:51:46,150][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:51:46,682][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:51:47,181][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:51:47,681][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:51:48,178][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:51:48,676][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:51:49,175][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:51:49,675][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:51:50,171][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:51:50,668][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:51:51,164][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:51:51,663][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:51:52,158][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:51:52,657][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:51:53,153][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:51:53,649][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:51:54,150][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:51:54,646][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:51:55,141][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:51:55,639][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:51:56,136][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:51:56,632][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:51:57,130][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:51:57,627][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:51:58,143][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:51:58,638][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:51:59,139][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:51:59,640][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:52:00,140][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:52:00,645][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:52:01,146][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:52:01,643][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:52:02,145][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:52:02,647][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:52:03,148][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:52:03,651][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:52:04,150][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:52:04,660][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:52:05,165][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:52:05,664][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:52:06,166][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:52:06,664][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:52:07,167][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:52:07,668][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:52:08,169][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:52:08,675][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:52:09,178][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:52:09,681][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:52:10,184][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:52:10,686][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:52:11,188][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:52:12,801][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:52:13,319][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:52:13,847][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10061 tokens.
[2025-11-13 05:52:14,765][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:34
[2025-11-13 05:52:15,458][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:52:15,459][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:52:15,461][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:52:16,380][__main__][INFO] - Iteration 497 took 1m 1s (39.17% Gen, 59.34% Train). Generation: 24s, Training: 36s. Estimated remaining time: 43h 32m 54s. Estimated total time: 51h 15m 28s. Time estimates for 10 more iterations: 10m 15s, 100 more iterations: 1h 42m 30s, 500 more iterations: 8h 32m 34s.
[2025-11-13 05:52:16,384][__main__][INFO] - Starting iteration 497.
[2025-11-13 05:52:16,867][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:52:16,867][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:52:40,256][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:52:52,864][__main__][INFO] - Number of regex retries in iteration 497: 1
[2025-11-13 05:52:52,864][__main__][INFO] - agents played in iteration 497 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:52:53,745][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:52:53,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:52:53,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:52:53,817][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:52:53,818][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:52:53,818][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:52:54,576][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:52:55,037][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:52:55,539][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:52:56,052][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:52:56,549][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:52:57,046][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:52:57,558][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:52:58,055][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:52:58,554][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:52:59,059][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:52:59,558][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:53:00,065][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:53:00,564][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:53:01,088][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:53:01,588][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:53:02,093][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:53:02,596][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:53:03,095][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:53:03,592][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:53:04,095][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:53:04,595][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:53:05,095][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:53:05,596][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:53:06,096][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:53:06,597][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:53:07,101][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:53:07,600][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:53:08,101][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:53:08,602][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:53:09,104][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:53:09,604][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:53:10,105][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:53:10,613][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:53:11,115][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:53:11,616][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:53:12,116][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:53:12,616][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:53:13,133][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:53:13,642][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:53:14,140][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:53:14,644][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:53:15,147][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:53:15,654][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:53:16,156][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:53:16,660][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:53:17,160][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:53:17,662][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:53:18,167][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:53:18,668][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:53:19,167][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:53:19,668][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:53:20,169][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:53:20,673][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:53:21,173][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:53:21,674][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:53:22,176][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:53:22,676][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:53:23,183][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:53:23,687][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:53:24,188][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:53:24,695][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:53:25,204][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:53:25,709][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:53:26,217][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:53:26,724][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10195 tokens.
[2025-11-13 05:53:27,605][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:33
[2025-11-13 05:53:28,402][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:53:28,404][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:53:28,406][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:53:29,434][__main__][INFO] - Iteration 498 took 1m 12s (49.60% Gen, 48.98% Train). Generation: 35s, Training: 35s. Estimated remaining time: 52h 44m 35s. Estimated total time: 60h 28m 23s. Time estimates for 10 more iterations: 12m 5s, 100 more iterations: 2h 0m 56s, 500 more iterations: 10h 4m 43s.
[2025-11-13 05:53:29,436][__main__][INFO] - Starting iteration 498.
[2025-11-13 05:53:29,922][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:53:29,922][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:53:55,315][__main__][INFO] - Number of regex retries in iteration 498: 0
[2025-11-13 05:53:55,317][__main__][INFO] - agents played in iteration 498 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:53:56,248][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:53:56,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:53:56,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:53:56,327][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:53:56,328][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:53:56,329][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:53:57,079][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:53:57,542][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:53:58,047][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:53:58,547][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:53:59,045][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:53:59,543][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:54:00,056][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:54:00,558][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:54:01,059][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:54:01,555][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:54:02,055][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:54:02,565][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:54:03,064][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:54:03,562][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:54:04,065][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:54:04,565][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:54:05,064][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:54:05,561][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:54:06,059][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:54:06,572][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:54:07,075][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:54:07,573][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:54:08,073][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:54:08,573][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:54:09,069][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:54:09,565][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:54:10,060][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:54:10,567][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:54:11,062][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:54:11,558][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:54:12,056][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:54:12,557][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:54:13,063][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:54:13,565][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:54:14,067][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:54:14,572][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:54:15,076][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:54:15,581][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:54:16,083][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:54:16,584][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:54:17,086][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:54:17,585][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:54:18,084][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:54:18,589][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:54:19,091][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:54:19,593][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:54:20,098][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:54:20,603][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:54:21,106][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:54:21,611][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:54:22,112][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:54:22,615][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:54:23,115][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:54:23,617][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:54:24,117][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:54:24,621][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:54:25,137][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:54:25,641][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:54:26,147][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:54:26,654][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:54:27,160][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:54:27,669][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:54:28,171][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:54:28,677][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:54:29,182][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10089 tokens.
[2025-11-13 05:54:30,073][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:33
[2025-11-13 05:54:30,740][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:54:30,742][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:54:30,744][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:54:31,669][__main__][INFO] - Iteration 499 took 1m 1s (41.13% Gen, 57.37% Train). Generation: 25s, Training: 35s. Estimated remaining time: 43h 42m 32s. Estimated total time: 51h 27m 22s. Time estimates for 10 more iterations: 10m 17s, 100 more iterations: 1h 42m 54s, 500 more iterations: 8h 34m 33s.
[2025-11-13 05:54:31,671][__main__][INFO] - Starting iteration 499.
[2025-11-13 05:54:32,149][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:54:32,149][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:55:03,600][__main__][INFO] - Number of regex retries in iteration 499: 0
[2025-11-13 05:55:03,601][__main__][INFO] - agents played in iteration 499 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:55:04,405][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:55:04,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:55:04,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:55:04,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:55:04,495][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:55:04,497][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:55:05,237][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:55:05,687][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:55:06,187][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:55:06,682][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:55:07,182][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:55:07,676][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:55:08,173][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:55:08,680][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:55:09,175][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:55:09,671][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:55:10,181][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:55:10,680][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:55:11,183][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:55:11,680][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:55:12,180][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:55:12,685][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:55:13,181][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:55:13,681][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:55:14,179][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:55:14,675][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:55:15,178][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:55:15,677][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:55:16,177][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:55:16,673][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:55:17,168][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:55:17,663][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:55:18,158][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:55:18,657][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:55:19,173][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:55:19,671][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:55:20,169][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:55:20,667][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:55:21,165][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:55:21,675][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:55:22,177][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:55:22,680][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:55:23,181][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:55:23,680][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:55:24,186][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:55:24,687][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:55:25,188][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:55:25,695][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:55:26,226][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:55:26,730][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:55:27,247][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:55:27,752][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:55:28,269][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:55:28,774][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:55:29,277][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:55:29,782][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:55:30,281][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:55:30,781][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:55:31,283][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:55:31,787][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:55:32,289][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:55:32,790][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:55:33,295][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:55:33,802][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:55:34,304][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:55:34,809][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:55:35,309][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:55:35,816][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:55:36,322][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:55:36,823][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:55:37,326][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10196 tokens.
[2025-11-13 05:55:38,237][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:33
[2025-11-13 05:55:38,953][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:55:38,955][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:55:38,956][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:55:39,996][__main__][INFO] - Iteration 500 took 1m 7s (46.36% Gen, 52.11% Train). Generation: 31s, Training: 35s. Estimated remaining time: 48h 46m 23s. Estimated total time: 56h 32m 22s. Time estimates for 10 more iterations: 11m 18s, 100 more iterations: 1h 53m 4s, 500 more iterations: 9h 25m 23s.
[2025-11-13 05:55:39,999][__main__][INFO] - Starting iteration 500.
[2025-11-13 05:55:40,483][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 05:55:40,484][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:55:58,900][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 05:56:10,580][__main__][INFO] - Number of regex retries in iteration 500: 1
[2025-11-13 05:56:10,581][__main__][INFO] - agents played in iteration 500 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:56:11,427][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:56:11,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:56:11,479][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:56:11,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:56:11,503][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:56:11,504][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:56:12,250][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:56:12,705][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:56:13,208][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:56:13,705][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:56:14,207][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:56:14,704][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:56:15,200][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:56:15,698][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:56:16,198][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:56:16,698][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:56:17,201][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:56:17,696][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:56:18,199][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:56:18,696][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:56:19,193][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:56:19,693][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:56:20,189][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:56:20,694][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:56:21,189][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:56:21,685][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:56:22,187][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:56:22,683][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:56:23,179][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:56:23,679][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:56:24,176][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:56:24,691][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:56:25,191][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:56:25,712][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:56:26,220][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:56:26,725][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:56:27,229][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:56:27,733][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:56:28,239][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:56:28,742][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:56:29,244][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:56:29,749][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:56:30,250][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:56:30,751][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:56:31,253][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:56:31,754][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:56:32,255][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:56:32,767][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:56:33,266][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:56:33,780][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:56:34,280][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:56:34,779][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:56:35,278][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:56:35,777][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:56:36,290][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:56:36,791][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:56:37,296][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:56:37,805][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:56:38,309][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:56:38,817][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:56:39,318][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:56:39,821][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:56:40,324][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:56:40,830][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:56:41,335][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:56:41,840][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:56:42,346][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:56:42,850][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:56:43,354][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:56:43,858][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:56:44,370][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10146 tokens.
[2025-11-13 05:56:45,292][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:33
[2025-11-13 05:56:45,938][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:56:45,940][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:56:45,943][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:56:47,783][__main__][INFO] - Iteration 501 took 1m 7s (44.72% Gen, 52.54% Train). Generation: 30s, Training: 35s. Estimated remaining time: 48h 17m 57s. Estimated total time: 56h 5m 3s. Time estimates for 10 more iterations: 11m 13s, 100 more iterations: 1h 52m 10s, 500 more iterations: 9h 20m 50s.
[2025-11-13 05:56:47,786][__main__][INFO] - Starting iteration 501.
[2025-11-13 05:56:48,272][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 05:56:48,273][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:57:15,328][__main__][INFO] - Number of regex retries in iteration 501: 0
[2025-11-13 05:57:15,329][__main__][INFO] - agents played in iteration 501 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:57:16,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:57:16,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:57:16,157][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:57:16,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:57:16,180][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:57:16,181][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:57:16,948][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:57:17,407][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:57:17,909][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:57:18,406][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:57:18,907][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:57:19,408][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:57:19,903][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:57:20,403][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:57:20,896][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:57:21,394][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:57:21,897][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:57:22,397][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:57:22,896][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:57:23,398][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:57:23,899][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:57:24,401][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:57:24,900][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:57:25,399][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:57:25,906][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:57:26,401][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:57:26,900][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:57:27,395][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:57:27,893][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:57:28,395][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:57:28,900][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:57:29,404][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:57:29,915][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:57:30,419][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:57:30,921][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:57:31,424][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:57:31,922][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:57:32,427][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:57:32,933][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:57:33,438][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:57:33,942][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:57:34,450][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:57:34,960][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:57:35,463][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:57:35,965][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:57:36,476][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:57:36,980][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:57:37,494][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:57:38,001][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:57:38,505][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:57:39,017][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:57:39,521][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:57:40,030][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:57:40,537][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:57:41,047][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:57:41,549][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:57:42,055][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:57:42,558][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:57:43,079][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:57:43,581][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:57:44,087][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:57:44,591][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:57:45,098][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:57:45,617][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:57:46,121][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:57:46,673][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:57:48,463][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:57:49,021][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:57:49,524][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:57:50,029][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:57:50,533][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10187 tokens.
[2025-11-13 05:57:51,450][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:34
[2025-11-13 05:57:52,127][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:57:52,129][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:57:52,132][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:57:53,066][__main__][INFO] - Iteration 502 took 1m 4s (41.76% Gen, 56.80% Train). Generation: 27s, Training: 36s. Estimated remaining time: 46h 11m 32s. Estimated total time: 53h 59m 43s. Time estimates for 10 more iterations: 10m 47s, 100 more iterations: 1h 47m 59s, 500 more iterations: 8h 59m 57s.
[2025-11-13 05:57:53,069][__main__][INFO] - Starting iteration 502.
[2025-11-13 05:57:53,556][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 05:57:53,557][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:58:25,355][__main__][INFO] - Number of regex retries in iteration 502: 0
[2025-11-13 05:58:25,356][__main__][INFO] - agents played in iteration 502 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:58:26,206][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:58:26,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:58:26,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:58:26,295][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:58:26,296][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:58:26,297][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:58:27,094][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:58:27,553][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:58:28,053][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:58:28,551][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:58:29,054][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:58:29,555][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:58:30,053][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:58:30,553][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:58:31,051][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:58:31,547][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:58:32,043][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:58:32,539][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:58:33,037][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:58:33,538][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:58:34,042][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:58:34,547][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:58:35,053][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:58:35,559][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:58:36,058][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:58:36,560][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:58:37,063][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:58:37,566][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:58:38,067][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:58:38,569][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:58:39,071][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:58:39,584][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:58:40,087][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:58:40,587][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:58:41,097][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:58:41,603][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:58:42,109][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:58:42,608][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:58:43,108][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:58:43,617][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:58:44,120][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:58:44,623][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:58:45,124][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:58:45,624][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:58:46,126][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:58:46,625][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:58:47,130][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:58:47,634][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:58:48,147][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:58:48,650][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:58:49,161][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:58:49,664][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:58:50,177][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:58:50,684][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:58:51,185][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:58:51,690][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:58:52,192][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:58:52,703][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:58:53,203][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:58:53,705][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:58:54,208][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:58:54,711][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:58:55,214][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:58:55,720][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:58:56,221][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:58:56,729][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:58:57,229][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:58:57,731][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:58:58,235][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:58:58,737][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 05:58:59,238][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10129 tokens.
[2025-11-13 05:59:00,093][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.06%, ΔTime: 00:00:33
[2025-11-13 05:59:00,891][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 05:59:00,892][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 05:59:00,894][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 05:59:02,005][__main__][INFO] - Iteration 503 took 1m 8s (46.45% Gen, 51.92% Train). Generation: 31s, Training: 35s. Estimated remaining time: 49h 13m 10s. Estimated total time: 57h 2m 30s. Time estimates for 10 more iterations: 11m 24s, 100 more iterations: 1h 54m 5s, 500 more iterations: 9h 30m 25s.
[2025-11-13 05:59:02,007][__main__][INFO] - Starting iteration 503.
[2025-11-13 05:59:02,492][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 05:59:02,492][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 05:59:25,672][__main__][INFO] - Number of regex retries in iteration 503: 0
[2025-11-13 05:59:25,673][__main__][INFO] - agents played in iteration 503 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 05:59:26,474][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:59:26,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:59:26,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:59:26,540][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 05:59:26,541][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 05:59:26,541][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 05:59:27,353][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 05:59:27,807][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 05:59:28,319][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 05:59:28,827][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 05:59:29,330][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 05:59:29,836][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 05:59:30,336][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 05:59:30,841][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 05:59:31,337][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 05:59:31,837][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 05:59:32,337][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 05:59:32,838][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 05:59:33,334][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 05:59:33,831][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 05:59:34,328][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 05:59:34,834][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 05:59:35,335][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 05:59:35,836][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 05:59:36,333][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 05:59:36,829][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 05:59:37,330][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 05:59:37,827][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 05:59:38,327][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 05:59:38,825][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 05:59:39,320][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 05:59:39,816][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 05:59:40,313][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 05:59:40,811][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 05:59:41,319][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 05:59:41,821][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 05:59:42,319][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 05:59:42,819][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 05:59:43,318][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 05:59:43,823][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 05:59:44,323][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 05:59:44,826][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 05:59:45,332][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 05:59:45,834][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 05:59:46,335][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 05:59:46,840][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 05:59:47,343][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 05:59:47,847][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 05:59:48,347][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 05:59:48,849][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 05:59:49,357][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 05:59:49,858][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 05:59:50,364][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 05:59:50,866][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 05:59:52,200][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 05:59:52,716][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 05:59:53,227][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 05:59:53,727][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 05:59:54,231][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 05:59:54,730][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 05:59:55,233][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 05:59:55,738][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 05:59:56,239][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 05:59:56,743][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 05:59:57,248][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 05:59:57,750][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 05:59:58,253][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 05:59:58,781][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 05:59:59,286][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 05:59:59,802][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:00:00,307][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10135 tokens.
[2025-11-13 06:00:01,207][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.28%, Current % of VRAM taken: 60.54%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:33
[2025-11-13 06:00:01,865][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:00:01,867][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:00:01,869][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:00:02,796][__main__][INFO] - Iteration 504 took 1m 0s (38.44% Gen, 60.02% Train). Generation: 23s, Training: 36s. Estimated remaining time: 42h 24m 52s. Estimated total time: 50h 15m 13s. Time estimates for 10 more iterations: 10m 3s, 100 more iterations: 1h 40m 30s, 500 more iterations: 8h 22m 32s.
[2025-11-13 06:00:02,800][__main__][INFO] - Starting iteration 504.
[2025-11-13 06:00:03,297][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 06:00:03,298][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:00:32,993][mllm.models.large_language_model_local][WARNING] - Response Given the values, it's clear that both hats and books have high value for Bob, and balls have high value for you. To maximize your points, you should propose to take all the balls and split the hats and books equally or in a way that maximizes your value while not overallocating.

Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:00:34,139][__main__][INFO] - Number of regex retries in iteration 504: 1
[2025-11-13 06:00:34,140][__main__][INFO] - agents played in iteration 504 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:00:35,128][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:00:35,154][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:00:35,181][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:00:35,204][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:00:35,204][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:00:35,206][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:00:35,994][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:00:36,449][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:00:36,960][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:00:37,458][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:00:37,964][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:00:38,468][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:00:38,967][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:00:39,470][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:00:39,971][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:00:40,467][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:00:40,965][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:00:41,461][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:00:41,959][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:00:42,459][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:00:42,955][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:00:43,456][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:00:43,956][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:00:44,457][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:00:44,954][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:00:45,453][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:00:45,957][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:00:46,462][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:00:46,963][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:00:47,465][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:00:47,966][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:00:48,462][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:00:48,960][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:00:49,459][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:00:49,960][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:00:50,459][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:00:50,958][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:00:51,460][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:00:51,959][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:00:52,462][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:00:52,963][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:00:53,464][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:00:53,971][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:00:54,471][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:00:54,973][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:00:55,475][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:00:55,981][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:00:56,484][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:00:56,986][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:00:57,486][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:00:58,000][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:00:58,500][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:00:59,007][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:00:59,508][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:01:00,009][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:01:00,529][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:01:01,031][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:01:01,533][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:01:02,032][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:01:02,536][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:01:03,042][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:01:03,543][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:01:04,044][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:01:04,549][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:01:05,052][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:01:05,555][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:01:06,058][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:01:06,560][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:01:07,063][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:01:07,565][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:01:08,068][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9981 tokens.
[2025-11-13 06:01:08,928][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:32
[2025-11-13 06:01:09,702][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:01:09,704][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:01:09,706][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:01:10,798][__main__][INFO] - Iteration 505 took 1m 7s (45.69% Gen, 52.69% Train). Generation: 30s, Training: 35s. Estimated remaining time: 48h 23m 36s. Estimated total time: 56h 15m 5s. Time estimates for 10 more iterations: 11m 15s, 100 more iterations: 1h 52m 30s, 500 more iterations: 9h 22m 30s.
[2025-11-13 06:01:10,800][__main__][INFO] - Starting iteration 505.
[2025-11-13 06:01:11,278][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 06:01:11,279][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:01:33,985][__main__][INFO] - Number of regex retries in iteration 505: 0
[2025-11-13 06:01:33,986][__main__][INFO] - agents played in iteration 505 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:01:34,945][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:01:34,967][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:01:34,990][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:01:35,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:01:35,012][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:01:35,013][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:01:35,873][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:01:36,332][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:01:36,838][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:01:37,345][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:01:37,846][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:01:38,343][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:01:38,844][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:01:39,346][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:01:39,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:01:40,339][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:01:40,837][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:01:41,342][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:01:41,839][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:01:42,344][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:01:42,838][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:01:43,341][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:01:43,845][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:01:44,342][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:01:44,841][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:01:45,350][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:01:45,847][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:01:46,348][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:01:46,843][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:01:47,341][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:01:47,844][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:01:48,344][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:01:48,840][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:01:49,340][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:01:49,836][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:01:50,337][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:01:50,838][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:01:51,333][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:01:51,846][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:01:52,346][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:01:52,846][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:01:53,344][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:01:53,842][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:01:54,346][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:01:54,844][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:01:55,345][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:01:55,854][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:01:56,358][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:01:56,863][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:01:57,363][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:01:57,864][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:01:58,366][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:01:58,867][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:01:59,367][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:01:59,867][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:02:00,368][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:02:00,875][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:02:01,377][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:02:01,878][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:02:02,381][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:02:03,711][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:02:05,314][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:02:05,819][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:02:06,324][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:02:06,824][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:02:07,339][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:02:07,844][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:02:08,343][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:02:08,848][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:02:09,347][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:02:09,850][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10100 tokens.
[2025-11-13 06:02:10,739][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:34
[2025-11-13 06:02:11,424][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:02:11,426][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:02:11,428][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:02:12,328][__main__][INFO] - Iteration 506 took 1m 1s (37.19% Gen, 61.33% Train). Generation: 22s, Training: 37s. Estimated remaining time: 43h 0m 0s. Estimated total time: 50h 52m 30s. Time estimates for 10 more iterations: 10m 10s, 100 more iterations: 1h 41m 45s, 500 more iterations: 8h 28m 45s.
[2025-11-13 06:02:12,331][__main__][INFO] - Starting iteration 506.
[2025-11-13 06:02:12,824][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 06:02:12,825][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:02:46,016][__main__][INFO] - Number of regex retries in iteration 506: 0
[2025-11-13 06:02:46,017][__main__][INFO] - agents played in iteration 506 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:02:46,869][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:02:46,896][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:02:46,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:02:46,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:02:46,947][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:02:46,948][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:02:47,698][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:02:48,156][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:02:48,657][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:02:49,159][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:02:49,658][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:02:50,154][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:02:50,666][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:02:51,165][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:02:51,663][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:02:52,161][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:02:52,657][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:02:53,166][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:02:53,661][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:02:54,158][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:02:54,657][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:02:55,158][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:02:55,659][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:02:56,158][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:02:56,660][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:02:57,181][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:02:57,682][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:02:58,189][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:02:58,692][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:02:59,195][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:02:59,699][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:03:00,199][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:03:00,698][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:03:01,204][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:03:01,703][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:03:02,203][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:03:02,703][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:03:03,208][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:03:03,710][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:03:04,213][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:03:04,718][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:03:05,244][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:03:05,756][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:03:06,259][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:03:06,763][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:03:07,269][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:03:07,783][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:03:08,284][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:03:08,793][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:03:09,296][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:03:09,798][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:03:10,304][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:03:10,804][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:03:11,305][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:03:11,807][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:03:12,310][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:03:12,814][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:03:13,317][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:03:13,819][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:03:14,323][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:03:14,824][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:03:15,326][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:03:15,829][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:03:16,329][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:03:16,834][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:03:17,336][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:03:17,840][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:03:18,342][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:03:18,844][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:03:19,344][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:03:19,845][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10105 tokens.
[2025-11-13 06:03:20,727][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:33
[2025-11-13 06:03:21,501][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:03:21,503][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:03:21,505][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:03:22,618][__main__][INFO] - Iteration 507 took 1m 9s (47.56% Gen, 50.85% Train). Generation: 33s, Training: 35s. Estimated remaining time: 50h 16m 0s. Estimated total time: 58h 9m 41s. Time estimates for 10 more iterations: 11m 37s, 100 more iterations: 1h 56m 19s, 500 more iterations: 9h 41m 36s.
[2025-11-13 06:03:22,620][__main__][INFO] - Starting iteration 507.
[2025-11-13 06:03:23,101][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 06:03:23,102][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:03:48,982][__main__][INFO] - Number of regex retries in iteration 507: 0
[2025-11-13 06:03:48,983][__main__][INFO] - agents played in iteration 507 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:03:49,765][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:03:49,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:03:49,814][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:03:49,837][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:03:49,837][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:03:49,838][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:03:50,583][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:03:51,040][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:03:51,558][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:03:52,057][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:03:52,579][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:03:53,080][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:03:53,577][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:03:54,075][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:03:54,573][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:03:55,077][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:03:55,572][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:03:56,068][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:03:56,585][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:03:57,081][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:03:57,577][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:03:58,077][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:03:58,575][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:03:59,077][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:03:59,576][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:04:00,072][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:04:00,585][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:04:01,082][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:04:01,582][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:04:02,081][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:04:02,578][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:04:03,082][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:04:03,579][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:04:04,076][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:04:04,583][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:04:05,083][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:04:05,583][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:04:06,084][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:04:06,584][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:04:07,085][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:04:07,583][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:04:08,086][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:04:08,593][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:04:09,155][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:04:10,814][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:04:11,324][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:04:11,827][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:04:12,329][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:04:12,831][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:04:13,335][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:04:13,839][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:04:14,342][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:04:14,845][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:04:15,358][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:04:15,859][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:04:16,383][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:04:16,887][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:04:17,390][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:04:17,897][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:04:18,401][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:04:18,906][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:04:19,409][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:04:19,913][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:04:20,417][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:04:20,919][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:04:21,422][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:04:21,928][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:04:22,428][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:04:22,931][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:04:23,436][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:04:23,937][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10157 tokens.
[2025-11-13 06:04:24,830][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:34
[2025-11-13 06:04:25,470][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:04:25,472][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:04:25,473][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:04:26,424][__main__][INFO] - Iteration 508 took 1m 3s (40.87% Gen, 57.63% Train). Generation: 25s, Training: 36s. Estimated remaining time: 44h 51m 25s. Estimated total time: 52h 46m 10s. Time estimates for 10 more iterations: 10m 33s, 100 more iterations: 1h 45m 32s, 500 more iterations: 8h 47m 41s.
[2025-11-13 06:04:26,426][__main__][INFO] - Starting iteration 508.
[2025-11-13 06:04:26,908][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 06:04:26,908][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:04:55,498][__main__][INFO] - Number of regex retries in iteration 508: 0
[2025-11-13 06:04:55,498][__main__][INFO] - agents played in iteration 508 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:04:56,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:04:56,505][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:04:56,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:04:56,556][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:04:56,557][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:04:56,558][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:04:57,311][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:04:57,764][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:04:58,276][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:04:58,795][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:04:59,292][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:04:59,796][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:05:00,296][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:05:00,795][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:05:01,311][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:05:01,808][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:05:02,310][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:05:02,804][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:05:03,301][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:05:03,803][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:05:04,304][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:05:04,800][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:05:05,310][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:05:05,805][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:05:06,301][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:05:06,798][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:05:07,299][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:05:07,807][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:05:08,307][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:05:08,805][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:05:09,302][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:05:09,799][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:05:10,304][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:05:10,800][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:05:11,298][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:05:11,797][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:05:12,295][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:05:12,803][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:05:13,299][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:05:13,799][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:05:14,299][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:05:14,799][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:05:15,301][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:05:15,803][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:05:16,301][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:05:16,814][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:05:17,314][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:05:17,814][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:05:18,314][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:05:18,817][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:05:19,319][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:05:19,822][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:05:20,327][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:05:20,832][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:05:21,339][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:05:21,840][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:05:22,340][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:05:22,843][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:05:23,349][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:05:23,851][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:05:24,355][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:05:24,861][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:05:25,362][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:05:25,863][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:05:26,372][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:05:26,876][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:05:27,397][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:05:27,900][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:05:28,402][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:05:28,909][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:05:29,412][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10159 tokens.
[2025-11-13 06:05:30,320][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:33
[2025-11-13 06:05:31,073][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:05:31,075][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:05:31,077][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:05:32,111][__main__][INFO] - Iteration 509 took 1m 5s (43.85% Gen, 54.57% Train). Generation: 28s, Training: 35s. Estimated remaining time: 46h 24m 22s. Estimated total time: 54h 20m 12s. Time estimates for 10 more iterations: 10m 52s, 100 more iterations: 1h 48m 40s, 500 more iterations: 9h 3m 22s.
[2025-11-13 06:05:32,113][__main__][INFO] - Starting iteration 509.
[2025-11-13 06:05:32,599][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 06:05:32,600][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:05:48,116][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:06:00,265][__main__][INFO] - Number of regex retries in iteration 509: 1
[2025-11-13 06:06:00,266][__main__][INFO] - agents played in iteration 509 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:06:01,106][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:06:01,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:06:01,160][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:06:01,183][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:06:01,183][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:06:01,184][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:06:01,938][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:06:02,396][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:06:02,897][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:06:03,394][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:06:03,891][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:06:04,387][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:06:04,882][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:06:05,391][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:06:05,885][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:06:06,388][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:06:06,887][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:06:07,386][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:06:07,887][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:06:08,385][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:06:08,884][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:06:09,398][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:06:09,895][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:06:10,393][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:06:10,895][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:06:11,393][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:06:11,900][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:06:12,401][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:06:12,897][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:06:13,400][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:06:13,897][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:06:14,397][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:06:14,895][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:06:15,396][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:06:15,912][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:06:16,407][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:06:16,908][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:06:17,405][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:06:17,908][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:06:18,419][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:06:18,920][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:06:19,421][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:06:19,925][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:06:20,428][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:06:20,934][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:06:21,437][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:06:21,939][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:06:22,442][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:06:22,942][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:06:23,446][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:06:23,946][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:06:24,447][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:06:24,949][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:06:25,450][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:06:25,954][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:06:26,460][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:06:26,970][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:06:27,474][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:06:27,976][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:06:28,481][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:06:28,987][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:06:29,491][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:06:29,995][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:06:30,509][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:06:31,010][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:06:31,522][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:06:32,023][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:06:32,524][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:06:33,028][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:06:33,529][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:06:34,034][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10126 tokens.
[2025-11-13 06:06:34,920][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 06:06:35,571][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:06:35,573][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:06:35,575][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:06:36,509][__main__][INFO] - Iteration 510 took 1m 3s (43.29% Gen, 55.25% Train). Generation: 27s, Training: 35s. Estimated remaining time: 45h 18m 38s. Estimated total time: 53h 15m 33s. Time estimates for 10 more iterations: 10m 39s, 100 more iterations: 1h 46m 31s, 500 more iterations: 8h 52m 35s.
[2025-11-13 06:06:36,512][__main__][INFO] - Starting iteration 510.
[2025-11-13 06:06:36,995][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 06:06:36,995][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:06:52,116][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:07:03,819][__main__][INFO] - Number of regex retries in iteration 510: 1
[2025-11-13 06:07:03,819][__main__][INFO] - agents played in iteration 510 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:07:04,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:07:04,688][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:07:04,713][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:07:04,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:07:04,736][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:07:04,736][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:07:05,491][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:07:05,946][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:07:06,447][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:07:06,948][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:07:07,451][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:07:07,946][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:07:08,444][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:07:08,944][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:07:09,443][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:07:09,941][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:07:10,439][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:07:10,945][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:07:11,447][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:07:11,946][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:07:12,445][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:07:12,941][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:07:13,444][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:07:13,944][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:07:14,442][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:07:14,946][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:07:15,443][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:07:15,941][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:07:16,440][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:07:16,941][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:07:17,448][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:07:17,949][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:07:18,446][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:07:18,947][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:07:19,445][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:07:19,945][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:07:20,442][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:07:20,941][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:07:21,446][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:07:21,949][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:07:22,448][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:07:22,943][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:07:23,440][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:07:23,942][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:07:24,442][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:07:24,946][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:07:25,454][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:07:25,955][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:07:26,458][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:07:26,962][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:07:27,467][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:07:27,968][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:07:28,470][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:07:28,976][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:07:29,506][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:07:30,014][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:07:30,521][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:07:31,026][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:07:31,530][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:07:32,038][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:07:32,545][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:07:33,049][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:07:33,554][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:07:34,059][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:07:34,563][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:07:35,065][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:07:35,570][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:07:36,073][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:07:36,578][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:07:37,081][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:07:37,584][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10117 tokens.
[2025-11-13 06:07:38,479][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 06:07:39,201][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:07:39,202][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:07:39,204][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:07:41,151][__main__][INFO] - Iteration 511 took 1m 4s (41.81% Gen, 55.15% Train). Generation: 26s, Training: 35s. Estimated remaining time: 45h 29m 52s. Estimated total time: 53h 27m 51s. Time estimates for 10 more iterations: 10m 41s, 100 more iterations: 1h 46m 55s, 500 more iterations: 8h 54m 38s.
[2025-11-13 06:07:41,154][__main__][INFO] - Starting iteration 511.
[2025-11-13 06:07:41,626][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:07:41,626][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:08:00,148][mllm.models.large_language_model_local][WARNING] - Response .Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:08:13,142][__main__][INFO] - Number of regex retries in iteration 511: 1
[2025-11-13 06:08:13,143][__main__][INFO] - agents played in iteration 511 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:08:14,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:08:14,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:08:14,064][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:08:14,087][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:08:14,087][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:08:14,088][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:08:14,853][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:08:15,312][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:08:15,813][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:08:16,310][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:08:16,808][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:08:17,305][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:08:17,802][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:08:18,302][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:08:18,800][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:08:19,300][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:08:19,798][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:08:20,296][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:08:20,797][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:08:21,299][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:08:21,799][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:08:22,298][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:08:22,796][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:08:23,295][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:08:23,796][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:08:24,295][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:08:24,803][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:08:25,302][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:08:25,800][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:08:26,301][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:08:26,798][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:08:27,300][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:08:27,798][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:08:28,301][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:08:28,806][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:08:29,321][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:08:29,823][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:08:30,325][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:08:30,830][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:08:31,355][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:08:31,860][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:08:32,363][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:08:32,870][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:08:33,376][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:08:33,880][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:08:34,382][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:08:34,892][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:08:35,392][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:08:35,894][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:08:36,397][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:08:36,903][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:08:37,406][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:08:37,912][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:08:38,415][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:08:38,925][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:08:39,437][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:08:39,942][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:08:40,455][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:08:40,961][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:08:41,465][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:08:41,967][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:08:42,471][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:08:42,978][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:08:43,479][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:08:43,982][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:08:44,486][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:08:44,996][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:08:45,499][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:08:46,003][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:08:46,508][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:08:47,013][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10211 tokens.
[2025-11-13 06:08:47,922][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:33
[2025-11-13 06:08:48,586][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:08:48,589][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:08:48,591][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:08:49,493][__main__][INFO] - Iteration 512 took 1m 7s (46.44% Gen, 52.23% Train). Generation: 31s, Training: 35s. Estimated remaining time: 48h 34m 16s. Estimated total time: 56h 33m 24s. Time estimates for 10 more iterations: 11m 18s, 100 more iterations: 1h 53m 6s, 500 more iterations: 9h 25m 34s.
[2025-11-13 06:08:49,496][__main__][INFO] - Starting iteration 512.
[2025-11-13 06:08:49,972][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:08:49,973][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:09:06,789][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:09:11,199][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:09:18,180][__main__][INFO] - Number of regex retries in iteration 512: 2
[2025-11-13 06:09:18,181][__main__][INFO] - agents played in iteration 512 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:09:19,074][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:09:19,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:09:19,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:09:19,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:09:19,142][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:09:19,144][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:09:19,902][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:09:20,356][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:09:20,859][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:09:21,366][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:09:21,865][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:09:22,363][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:09:22,860][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:09:23,361][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:09:23,865][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:09:24,365][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:09:24,865][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:09:25,370][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:09:25,869][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:09:26,369][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:09:26,867][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:09:27,364][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:09:27,870][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:09:28,365][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:09:28,887][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:09:29,383][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:09:29,882][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:09:30,388][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:09:30,887][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:09:31,382][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:09:31,880][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:09:32,385][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:09:32,888][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:09:33,392][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:09:33,893][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:09:34,397][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:09:34,897][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:09:35,399][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:09:35,907][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:09:36,413][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:09:36,915][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:09:37,427][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:09:37,962][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:09:38,477][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:09:38,981][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:09:39,484][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:09:39,985][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:09:40,491][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:09:41,000][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:09:41,504][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:09:42,008][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:09:42,516][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:09:43,021][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:09:43,527][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:09:44,034][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:09:44,538][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:09:45,057][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:09:45,559][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:09:46,064][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:09:46,568][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:09:47,070][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:09:47,577][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:09:48,556][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:09:49,998][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:09:50,501][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:09:51,008][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:09:51,517][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:09:52,022][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:09:52,526][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:09:53,040][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:09:53,545][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10178 tokens.
[2025-11-13 06:09:54,457][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:34
[2025-11-13 06:09:55,125][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:09:55,126][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:09:55,128][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:09:56,281][__main__][INFO] - Iteration 513 took 1m 6s (42.54% Gen, 55.72% Train). Generation: 28s, Training: 36s. Estimated remaining time: 47h 15m 15s. Estimated total time: 55h 15m 29s. Time estimates for 10 more iterations: 11m 3s, 100 more iterations: 1h 50m 30s, 500 more iterations: 9h 12m 34s.
[2025-11-13 06:09:56,284][__main__][INFO] - Starting iteration 513.
[2025-11-13 06:09:56,768][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:09:56,768][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:10:24,917][__main__][INFO] - Number of regex retries in iteration 513: 0
[2025-11-13 06:10:24,918][__main__][INFO] - agents played in iteration 513 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:10:25,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:10:25,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:10:25,819][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:10:25,842][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:10:25,842][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:10:25,843][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:10:26,596][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:10:27,054][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:10:27,558][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:10:28,065][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:10:28,563][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:10:29,063][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:10:29,570][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:10:30,070][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:10:30,570][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:10:31,079][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:10:31,579][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:10:32,081][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:10:32,579][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:10:33,080][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:10:33,585][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:10:34,089][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:10:34,589][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:10:35,091][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:10:35,597][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:10:36,099][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:10:36,599][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:10:37,104][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:10:37,607][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:10:38,112][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:10:38,619][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:10:39,123][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:10:39,626][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:10:40,133][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:10:40,634][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:10:41,139][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:10:41,658][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:10:42,160][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:10:42,663][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:10:43,172][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:10:43,672][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:10:44,172][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:10:44,676][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:10:45,181][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:10:45,689][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:10:46,190][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:10:46,690][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:10:47,193][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:10:47,695][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:10:48,202][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:10:48,705][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:10:49,209][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:10:49,722][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:10:50,224][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:10:50,725][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:10:51,229][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:10:51,736][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:10:52,250][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:10:52,756][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:10:53,259][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:10:53,768][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:10:54,269][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:10:54,781][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:10:55,284][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:10:55,788][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:10:56,292][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:10:56,798][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:10:57,300][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:10:57,805][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:10:58,307][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:10:58,814][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10215 tokens.
[2025-11-13 06:10:59,672][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.13%, ΔTime: 00:00:33
[2025-11-13 06:11:00,435][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:11:00,436][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:11:00,438][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:11:01,467][__main__][INFO] - Iteration 514 took 1m 4s (43.51% Gen, 54.90% Train). Generation: 28s, Training: 35s. Estimated remaining time: 45h 53m 40s. Estimated total time: 53h 55m 0s. Time estimates for 10 more iterations: 10m 47s, 100 more iterations: 1h 47m 50s, 500 more iterations: 8h 59m 10s.
[2025-11-13 06:11:01,469][__main__][INFO] - Starting iteration 514.
[2025-11-13 06:11:01,967][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:11:01,969][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:11:16,071][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:11:27,697][__main__][INFO] - Number of regex retries in iteration 514: 1
[2025-11-13 06:11:27,698][__main__][INFO] - agents played in iteration 514 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:11:28,536][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:11:28,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:11:28,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:11:28,616][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:11:28,617][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:11:28,617][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:11:29,373][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:11:29,826][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:11:30,335][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:11:30,837][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:11:31,335][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:11:31,837][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:11:32,339][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:11:32,836][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:11:33,335][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:11:33,832][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:11:34,334][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:11:34,833][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:11:35,332][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:11:35,832][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:11:36,329][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:11:36,825][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:11:37,322][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:11:37,822][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:11:38,320][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:11:38,817][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:11:39,315][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:11:39,819][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:11:40,317][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:11:40,816][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:11:41,317][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:11:41,819][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:11:42,321][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:11:42,823][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:11:43,326][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:11:43,830][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:11:44,337][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:11:44,841][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:11:45,368][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:11:45,868][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:11:46,371][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:11:46,880][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:11:47,382][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:11:47,884][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:11:48,385][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:11:48,907][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:11:49,407][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:11:49,910][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:11:50,412][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:11:50,919][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:11:51,422][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:11:51,927][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:11:52,431][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:11:52,941][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:11:53,442][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:11:53,945][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:11:54,455][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:11:54,960][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:11:55,468][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:11:55,971][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:11:56,477][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:11:56,995][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:11:57,503][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:11:59,072][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:11:59,638][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:12:00,153][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:12:00,653][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:12:01,168][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:12:01,671][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:12:02,193][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:12:02,693][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10108 tokens.
[2025-11-13 06:12:03,622][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:34
[2025-11-13 06:12:04,293][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:12:04,295][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:12:04,297][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:12:05,234][__main__][INFO] - Iteration 515 took 1m 3s (40.67% Gen, 57.85% Train). Generation: 25s, Training: 36s. Estimated remaining time: 44h 41m 0s. Estimated total time: 52h 43m 24s. Time estimates for 10 more iterations: 10m 32s, 100 more iterations: 1h 45m 26s, 500 more iterations: 8h 47m 14s.
[2025-11-13 06:12:05,238][__main__][INFO] - Starting iteration 515.
[2025-11-13 06:12:05,716][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:12:05,716][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:12:42,669][__main__][INFO] - Number of regex retries in iteration 515: 0
[2025-11-13 06:12:42,670][__main__][INFO] - agents played in iteration 515 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:12:43,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:12:43,558][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:12:43,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:12:43,609][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:12:43,609][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:12:43,610][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:12:44,429][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:12:44,883][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:12:45,386][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:12:45,903][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:12:46,403][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:12:46,902][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:12:47,403][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:12:47,903][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:12:48,403][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:12:48,907][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:12:49,410][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:12:49,918][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:12:50,421][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:12:50,925][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:12:51,427][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:12:51,935][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:12:52,438][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:12:52,968][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:12:53,475][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:12:53,988][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:12:54,491][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:12:55,005][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:12:55,509][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:12:56,017][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:12:56,526][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:12:57,033][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:12:57,537][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:12:58,038][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:12:58,543][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:12:59,045][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:12:59,546][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:13:00,048][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:13:00,552][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:13:01,058][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:13:01,562][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:13:02,065][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:13:02,568][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:13:03,072][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:13:03,577][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:13:04,082][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:13:04,587][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:13:05,086][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:13:05,600][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:13:06,102][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:13:06,602][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:13:07,103][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:13:07,604][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:13:08,114][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:13:08,615][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:13:09,118][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:13:09,622][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:13:10,123][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:13:10,626][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:13:11,129][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:13:11,631][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:13:12,138][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:13:12,640][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:13:13,146][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:13:13,645][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:13:14,146][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:13:14,648][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:13:15,147][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:13:15,647][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:13:16,151][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:13:16,655][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10151 tokens.
[2025-11-13 06:13:17,548][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:33
[2025-11-13 06:13:18,317][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:13:18,318][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:13:18,320][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:13:19,392][__main__][INFO] - Iteration 516 took 1m 13s (50.16% Gen, 48.39% Train). Generation: 36s, Training: 35s. Estimated remaining time: 53h 20m 13s. Estimated total time: 61h 23m 51s. Time estimates for 10 more iterations: 12m 16s, 100 more iterations: 2h 2m 47s, 500 more iterations: 10h 13m 58s.
[2025-11-13 06:13:19,394][__main__][INFO] - Starting iteration 516.
[2025-11-13 06:13:19,882][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:13:19,882][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:13:43,487][__main__][INFO] - Number of regex retries in iteration 516: 0
[2025-11-13 06:13:43,488][__main__][INFO] - agents played in iteration 516 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:13:44,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:13:44,405][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:13:44,430][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:13:44,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:13:44,453][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:13:44,454][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:13:45,255][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:13:45,712][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:13:46,214][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:13:46,725][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:13:47,224][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:13:47,723][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:13:48,221][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:13:48,720][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:13:49,226][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:13:49,725][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:13:50,226][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:13:50,722][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:13:51,220][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:13:51,723][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:13:52,220][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:13:52,722][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:13:53,227][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:13:53,730][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:13:54,229][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:13:54,736][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:13:55,240][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:13:55,837][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:13:56,342][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:13:56,845][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:13:57,347][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:13:57,849][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:13:58,351][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:13:58,851][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:13:59,351][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:13:59,853][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:14:00,359][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:14:00,861][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:14:01,362][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:14:01,863][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:14:02,367][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:14:02,869][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:14:03,375][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:14:03,878][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:14:04,376][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:14:04,880][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:14:05,386][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:14:05,889][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:14:06,482][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:14:08,464][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:14:08,979][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:14:09,481][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:14:09,989][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:14:10,493][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:14:10,996][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:14:11,517][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:14:12,023][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:14:12,530][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:14:13,027][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:14:13,531][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:14:14,034][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:14:14,539][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:14:15,045][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:14:15,549][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:14:16,051][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:14:16,553][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:14:17,053][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:14:17,563][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:14:18,069][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:14:18,572][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:14:19,077][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10046 tokens.
[2025-11-13 06:14:19,988][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:34
[2025-11-13 06:14:20,636][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:14:20,638][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:14:20,639][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:14:21,551][__main__][INFO] - Iteration 517 took 1m 1s (38.28% Gen, 60.24% Train). Generation: 23s, Training: 37s. Estimated remaining time: 43h 18m 47s. Estimated total time: 51h 23m 27s. Time estimates for 10 more iterations: 10m 16s, 100 more iterations: 1h 42m 46s, 500 more iterations: 8h 33m 54s.
[2025-11-13 06:14:21,553][__main__][INFO] - Starting iteration 517.
[2025-11-13 06:14:22,042][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:14:22,043][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:14:56,796][__main__][INFO] - Number of regex retries in iteration 517: 0
[2025-11-13 06:14:56,796][__main__][INFO] - agents played in iteration 517 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:14:57,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:14:57,697][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:14:57,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:14:57,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:14:57,758][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:14:57,759][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:14:58,549][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:14:59,005][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:14:59,519][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:15:00,023][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:15:00,527][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:15:01,027][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:15:01,527][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:15:02,046][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:15:02,547][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:15:03,048][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:15:03,550][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:15:04,051][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:15:04,562][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:15:05,065][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:15:05,564][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:15:06,068][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:15:06,568][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:15:07,075][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:15:07,578][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:15:08,078][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:15:08,582][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:15:09,088][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:15:09,588][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:15:10,091][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:15:10,594][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:15:11,098][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:15:11,611][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:15:12,114][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:15:12,637][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:15:13,145][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:15:13,645][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:15:14,155][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:15:14,658][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:15:15,159][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:15:15,660][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:15:16,163][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:15:16,665][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:15:17,168][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:15:17,669][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:15:18,170][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:15:18,672][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:15:19,200][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:15:19,703][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:15:20,204][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:15:20,719][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:15:21,222][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:15:21,735][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:15:22,236][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:15:22,742][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:15:23,252][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:15:23,751][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:15:24,250][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:15:24,750][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:15:25,252][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:15:25,757][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:15:26,263][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:15:26,763][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:15:27,266][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:15:27,770][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:15:28,271][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:15:28,774][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:15:29,282][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:15:29,785][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:15:30,288][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:15:30,789][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10083 tokens.
[2025-11-13 06:15:31,696][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:33
[2025-11-13 06:15:32,465][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:15:32,467][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:15:32,470][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:15:33,531][__main__][INFO] - Iteration 518 took 1m 11s (48.61% Gen, 49.90% Train). Generation: 34s, Training: 35s. Estimated remaining time: 51h 28m 36s. Estimated total time: 59h 34m 28s. Time estimates for 10 more iterations: 11m 54s, 100 more iterations: 1h 59m 8s, 500 more iterations: 9h 55m 44s.
[2025-11-13 06:15:33,533][__main__][INFO] - Starting iteration 518.
[2025-11-13 06:15:34,024][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:15:34,025][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:15:59,226][__main__][INFO] - Number of regex retries in iteration 518: 0
[2025-11-13 06:15:59,227][__main__][INFO] - agents played in iteration 518 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:16:00,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:16:00,028][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:16:00,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:16:00,076][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:16:00,077][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:16:00,077][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:16:00,832][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:16:01,292][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:16:01,796][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:16:02,294][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:16:02,794][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:16:03,293][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:16:03,799][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:16:04,302][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:16:04,798][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:16:05,320][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:16:05,820][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:16:06,322][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:16:06,822][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:16:07,325][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:16:07,840][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:16:08,343][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:16:08,852][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:16:09,351][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:16:09,854][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:16:10,360][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:16:10,861][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:16:11,360][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:16:11,863][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:16:12,363][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:16:12,904][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:16:15,138][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:16:15,651][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:16:16,158][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:16:16,661][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:16:17,168][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:16:17,668][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:16:18,170][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:16:18,676][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:16:19,179][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:16:19,684][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:16:20,189][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:16:20,689][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:16:21,190][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:16:21,695][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:16:22,198][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:16:22,702][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:16:23,209][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:16:23,715][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:16:24,217][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:16:24,724][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:16:25,228][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:16:25,738][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:16:26,257][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:16:26,762][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:16:27,265][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:16:27,764][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:16:28,265][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:16:28,774][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:16:29,280][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:16:29,784][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:16:30,288][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:16:30,790][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:16:31,294][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:16:31,795][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:16:32,302][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:16:32,809][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:16:33,310][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:16:33,817][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:16:34,332][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:16:34,837][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10127 tokens.
[2025-11-13 06:16:35,767][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:34
[2025-11-13 06:16:36,428][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:16:36,430][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:16:36,432][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:16:37,464][__main__][INFO] - Iteration 519 took 1m 3s (39.73% Gen, 58.64% Train). Generation: 25s, Training: 37s. Estimated remaining time: 44h 45m 7s. Estimated total time: 52h 52m 3s. Time estimates for 10 more iterations: 10m 34s, 100 more iterations: 1h 45m 44s, 500 more iterations: 8h 48m 40s.
[2025-11-13 06:16:37,468][__main__][INFO] - Starting iteration 519.
[2025-11-13 06:16:37,957][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:16:37,957][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:17:14,316][__main__][INFO] - Number of regex retries in iteration 519: 0
[2025-11-13 06:17:14,316][__main__][INFO] - agents played in iteration 519 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:17:15,191][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:17:15,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:17:15,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:17:15,267][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:17:15,268][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:17:15,268][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:17:16,113][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:17:16,571][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:17:17,081][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:17:17,581][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:17:18,083][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:17:18,585][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:17:19,089][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:17:19,593][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:17:20,101][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:17:20,601][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:17:21,100][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:17:21,602][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:17:22,103][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:17:22,604][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:17:23,103][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:17:23,606][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:17:24,105][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:17:24,603][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:17:25,113][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:17:25,612][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:17:26,114][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:17:26,617][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:17:27,118][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:17:27,625][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:17:28,127][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:17:28,629][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:17:29,138][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:17:29,644][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:17:30,160][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:17:30,662][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:17:31,165][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:17:31,680][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:17:32,211][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:17:32,711][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:17:33,214][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:17:33,717][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:17:34,223][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:17:34,721][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:17:35,223][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:17:35,725][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:17:36,229][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:17:36,731][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:17:37,233][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:17:37,733][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:17:38,234][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:17:38,733][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:17:39,240][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:17:39,752][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:17:40,253][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:17:40,767][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:17:41,275][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:17:41,779][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:17:42,285][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:17:42,789][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:17:43,293][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:17:43,798][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:17:44,302][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:17:44,807][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:17:45,307][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:17:45,810][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:17:46,315][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:17:46,819][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:17:47,321][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:17:47,824][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:17:48,326][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10080 tokens.
[2025-11-13 06:17:49,204][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:33
[2025-11-13 06:17:49,947][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:17:49,949][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:17:49,950][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:17:51,041][__main__][INFO] - Iteration 520 took 1m 13s (49.75% Gen, 48.76% Train). Generation: 36s, Training: 35s. Estimated remaining time: 52h 46m 5s. Estimated total time: 60h 54m 14s. Time estimates for 10 more iterations: 12m 10s, 100 more iterations: 2h 1m 48s, 500 more iterations: 10h 9m 2s.
[2025-11-13 06:17:51,043][__main__][INFO] - Starting iteration 520.
[2025-11-13 06:17:51,536][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 06:17:51,536][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:18:21,805][__main__][INFO] - Number of regex retries in iteration 520: 0
[2025-11-13 06:18:21,808][__main__][INFO] - agents played in iteration 520 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:18:22,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:18:22,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:18:22,731][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:18:22,755][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:18:22,756][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:18:22,757][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:18:23,633][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:18:24,091][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:18:24,607][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:18:25,112][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:18:25,614][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:18:26,121][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:18:26,623][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:18:27,127][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:18:27,631][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:18:28,131][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:18:28,639][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:18:29,141][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:18:29,643][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:18:30,148][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:18:30,649][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:18:31,154][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:18:31,654][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:18:32,158][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:18:32,671][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:18:33,173][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:18:33,677][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:18:34,187][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:18:34,710][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:18:35,216][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:18:35,727][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:18:36,233][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:18:36,740][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:18:37,244][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:18:37,747][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:18:38,251][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:18:38,759][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:18:39,270][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:18:39,775][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:18:40,282][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:18:40,785][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:18:41,290][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:18:41,794][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:18:42,300][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:18:42,808][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:18:43,313][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:18:43,815][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:18:44,323][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:18:44,824][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:18:45,326][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:18:45,830][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:18:46,331][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:18:46,834][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:18:47,334][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:18:47,835][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:18:48,339][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:18:48,841][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:18:49,341][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:18:49,842][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:18:50,344][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:18:50,846][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:18:51,346][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:18:51,848][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:18:52,351][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:18:52,853][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:18:53,355][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:18:53,860][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:18:54,361][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:18:54,863][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:18:55,370][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:18:55,871][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10117 tokens.
[2025-11-13 06:18:56,793][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 06:18:57,442][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:18:57,444][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:18:57,446][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:18:59,419][__main__][INFO] - Iteration 521 took 1m 7s (44.59% Gen, 52.50% Train). Generation: 30s, Training: 35s. Estimated remaining time: 48h 24m 57s. Estimated total time: 56h 34m 14s. Time estimates for 10 more iterations: 11m 18s, 100 more iterations: 1h 53m 8s, 500 more iterations: 9h 25m 42s.
[2025-11-13 06:18:59,422][__main__][INFO] - Starting iteration 521.
[2025-11-13 06:18:59,897][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:18:59,898][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:19:26,167][__main__][INFO] - Number of regex retries in iteration 521: 0
[2025-11-13 06:19:26,168][__main__][INFO] - agents played in iteration 521 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:19:26,942][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:19:26,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:19:27,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:19:27,025][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:19:27,025][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:19:27,026][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:19:27,797][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:19:28,250][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:19:28,757][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:19:29,260][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:19:29,761][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:19:30,266][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:19:30,774][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:19:31,275][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:19:31,778][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:19:32,276][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:19:32,791][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:19:33,290][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:19:33,789][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:19:34,293][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:19:34,795][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:19:35,297][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:19:35,796][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:19:36,296][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:19:36,814][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:19:37,315][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:19:37,820][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:19:38,327][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:19:38,828][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:19:39,334][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:19:39,835][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:19:40,337][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:19:40,843][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:19:41,348][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:19:41,848][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:19:42,352][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:19:42,851][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:19:43,353][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:19:43,858][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:19:44,359][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:19:44,863][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:19:45,366][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:19:45,866][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:19:46,366][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:19:46,867][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:19:47,371][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:19:47,874][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:19:48,374][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:19:48,878][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:19:49,378][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:19:49,885][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:19:50,383][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:19:50,881][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:19:51,389][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:19:51,888][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:19:52,392][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:19:52,893][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:19:53,394][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:19:53,909][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:19:54,409][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:19:56,334][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:19:57,138][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:19:57,642][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:19:58,143][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:19:58,645][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:19:59,151][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:19:59,661][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:20:00,164][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:20:00,669][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:20:01,168][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:20:01,666][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10038 tokens.
[2025-11-13 06:20:02,550][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 62.01%, ΔTime: 00:00:34
[2025-11-13 06:20:03,243][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:20:03,246][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:20:03,248][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:20:04,222][__main__][INFO] - Iteration 522 took 1m 4s (40.84% Gen, 57.64% Train). Generation: 26s, Training: 37s. Estimated remaining time: 45h 25m 54s. Estimated total time: 53h 36m 17s. Time estimates for 10 more iterations: 10m 43s, 100 more iterations: 1h 47m 12s, 500 more iterations: 8h 56m 2s.
[2025-11-13 06:20:04,225][__main__][INFO] - Starting iteration 522.
[2025-11-13 06:20:04,737][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:20:04,737][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:20:26,997][mllm.models.large_language_model_local][WARNING] - Response Proposal: x hats, y books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:20:39,110][__main__][INFO] - Number of regex retries in iteration 522: 1
[2025-11-13 06:20:39,111][__main__][INFO] - agents played in iteration 522 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:20:39,974][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:20:40,004][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:20:40,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:20:40,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:20:40,067][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:20:40,068][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:20:40,929][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:20:41,391][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:20:41,895][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:20:42,404][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:20:42,906][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:20:43,410][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:20:43,915][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:20:44,417][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:20:44,916][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:20:45,433][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:20:45,935][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:20:46,438][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:20:46,942][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:20:47,448][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:20:47,951][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:20:48,456][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:20:48,965][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:20:49,467][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:20:49,966][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:20:50,467][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:20:50,967][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:20:51,467][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:20:51,971][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:20:52,473][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:20:52,973][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:20:53,478][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:20:53,982][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:20:54,486][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:20:54,988][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:20:55,492][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:20:56,003][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:20:56,507][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:20:57,012][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:20:57,524][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:20:58,025][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:20:58,541][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:20:59,042][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:20:59,547][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:21:00,050][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:21:00,551][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:21:01,051][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:21:01,552][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:21:02,054][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:21:02,558][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:21:03,062][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:21:03,562][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:21:04,063][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:21:04,564][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:21:05,064][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:21:05,568][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:21:06,068][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:21:06,571][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:21:07,071][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:21:07,573][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:21:08,079][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:21:08,579][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:21:09,083][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:21:09,592][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:21:10,094][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:21:10,601][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:21:11,107][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:21:11,609][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:21:12,116][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:21:12,623][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:21:13,145][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10124 tokens.
[2025-11-13 06:21:13,991][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:33
[2025-11-13 06:21:14,760][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:21:14,762][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:21:14,764][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:21:15,783][__main__][INFO] - Iteration 523 took 1m 11s (48.38% Gen, 50.18% Train). Generation: 34s, Training: 35s. Estimated remaining time: 51h 0m 45s. Estimated total time: 59h 12m 19s. Time estimates for 10 more iterations: 11m 50s, 100 more iterations: 1h 58m 24s, 500 more iterations: 9h 52m 3s.
[2025-11-13 06:21:15,785][__main__][INFO] - Starting iteration 523.
[2025-11-13 06:21:16,258][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:21:16,259][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:21:31,535][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:21:41,212][__main__][INFO] - Number of regex retries in iteration 523: 1
[2025-11-13 06:21:41,213][__main__][INFO] - agents played in iteration 523 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:21:41,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:21:42,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:21:42,048][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:21:42,071][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:21:42,072][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:21:42,073][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:21:42,820][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:21:43,275][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:21:43,779][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:21:44,281][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:21:44,790][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:21:45,288][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:21:45,791][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:21:46,292][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:21:46,796][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:21:47,299][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:21:47,805][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:21:48,308][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:21:48,813][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:21:49,315][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:21:49,819][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:21:50,322][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:21:50,827][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:21:51,346][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:21:51,850][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:21:52,359][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:21:52,863][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:21:53,367][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:21:53,874][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:21:54,380][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:21:54,885][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:21:55,440][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:21:57,653][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:21:58,163][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:21:58,667][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:21:59,176][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:21:59,687][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:22:00,191][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:22:00,696][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:22:01,199][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:22:01,702][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:22:02,205][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:22:02,710][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:22:03,211][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:22:03,711][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:22:04,219][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:22:04,734][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:22:05,241][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:22:05,753][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:22:06,260][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:22:06,764][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:22:07,271][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:22:07,775][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:22:08,282][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:22:08,787][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:22:09,290][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:22:09,810][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:22:10,312][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:22:10,823][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:22:11,334][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:22:11,840][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:22:12,354][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:22:12,859][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:22:13,361][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:22:13,867][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:22:14,373][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:22:14,882][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:22:15,387][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:22:15,889][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:22:16,393][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:22:16,895][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10073 tokens.
[2025-11-13 06:22:17,791][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:34
[2025-11-13 06:22:18,434][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:22:18,436][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:22:18,438][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:22:19,424][__main__][INFO] - Iteration 524 took 1m 3s (39.50% Gen, 58.93% Train). Generation: 24s, Training: 37s. Estimated remaining time: 44h 25m 41s. Estimated total time: 52h 38m 19s. Time estimates for 10 more iterations: 10m 31s, 100 more iterations: 1h 45m 16s, 500 more iterations: 8h 46m 23s.
[2025-11-13 06:22:19,427][__main__][INFO] - Starting iteration 524.
[2025-11-13 06:22:19,916][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:22:19,917][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:22:51,765][__main__][INFO] - Number of regex retries in iteration 524: 0
[2025-11-13 06:22:51,766][__main__][INFO] - agents played in iteration 524 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:22:52,582][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:22:52,609][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:22:52,633][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:22:52,656][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:22:52,657][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:22:52,657][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:22:53,542][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:22:54,001][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:22:54,506][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:22:55,010][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:22:55,510][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:22:56,016][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:22:56,517][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:22:57,021][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:22:57,533][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:22:58,042][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:22:58,562][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:22:59,067][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:22:59,568][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:23:00,104][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:23:00,612][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:23:01,114][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:23:01,616][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:23:02,128][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:23:02,645][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:23:03,147][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:23:03,651][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:23:04,156][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:23:04,662][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:23:05,185][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:23:05,713][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:23:06,220][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:23:06,725][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:23:07,224][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:23:07,727][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:23:08,227][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:23:08,727][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:23:09,238][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:23:09,741][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:23:10,245][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:23:10,750][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:23:11,248][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:23:11,748][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:23:12,249][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:23:12,747][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:23:13,246][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:23:13,745][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:23:14,264][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:23:14,764][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:23:15,268][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:23:15,771][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:23:16,274][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:23:16,781][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:23:17,288][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:23:17,793][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:23:18,298][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:23:18,798][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:23:19,303][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:23:19,805][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:23:20,310][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:23:20,813][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:23:21,313][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:23:21,817][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:23:22,322][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:23:22,827][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:23:23,333][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:23:23,840][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:23:24,341][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:23:24,853][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:23:25,353][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:23:25,856][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10124 tokens.
[2025-11-13 06:23:26,706][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.44%, ΔTime: 00:00:33
[2025-11-13 06:23:27,475][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:23:27,476][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:23:27,478][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:23:28,481][__main__][INFO] - Iteration 525 took 1m 8s (46.45% Gen, 52.09% Train). Generation: 31s, Training: 35s. Estimated remaining time: 48h 54m 28s. Estimated total time: 57h 8m 14s. Time estimates for 10 more iterations: 11m 25s, 100 more iterations: 1h 54m 16s, 500 more iterations: 9h 31m 22s.
[2025-11-13 06:23:28,484][__main__][INFO] - Starting iteration 525.
[2025-11-13 06:23:28,961][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:23:28,962][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:23:44,674][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:23:47,598][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:23:55,374][__main__][INFO] - Number of regex retries in iteration 525: 2
[2025-11-13 06:23:55,375][__main__][INFO] - agents played in iteration 525 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:23:56,291][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:23:56,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:23:56,345][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:23:56,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:23:56,369][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:23:56,369][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:23:57,222][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:23:57,680][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:23:58,186][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:23:58,699][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:23:59,202][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:23:59,713][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:24:00,211][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:24:01,753][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:24:02,821][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:24:03,325][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:24:03,835][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:24:04,338][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:24:04,841][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:24:05,345][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:24:05,855][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:24:06,360][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:24:06,866][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:24:07,370][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:24:07,878][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:24:08,382][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:24:08,886][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:24:09,391][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:24:09,893][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:24:10,402][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:24:10,903][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:24:11,405][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:24:11,908][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:24:12,414][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:24:12,924][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:24:13,429][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:24:13,937][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:24:14,448][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:24:14,950][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:24:15,454][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:24:15,961][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:24:16,463][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:24:16,973][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:24:17,476][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:24:17,980][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:24:18,521][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:24:19,023][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:24:19,524][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:24:20,025][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:24:20,526][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:24:21,029][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:24:21,530][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:24:22,033][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:24:22,534][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:24:23,033][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:24:23,534][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:24:24,035][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:24:24,538][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:24:25,042][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:24:25,551][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:24:26,059][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:24:26,568][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:24:27,073][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:24:27,593][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:24:28,095][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:24:28,600][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:24:29,105][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:24:29,612][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:24:30,115][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:24:30,620][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:24:31,126][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10127 tokens.
[2025-11-13 06:24:32,103][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:34
[2025-11-13 06:24:32,768][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:24:32,770][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:24:32,773][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:24:33,819][__main__][INFO] - Iteration 526 took 1m 4s (40.72% Gen, 57.66% Train). Generation: 26s, Training: 37s. Estimated remaining time: 45h 48m 3s. Estimated total time: 54h 2m 55s. Time estimates for 10 more iterations: 10m 48s, 100 more iterations: 1h 48m 5s, 500 more iterations: 9h 0m 29s.
[2025-11-13 06:24:33,821][__main__][INFO] - Starting iteration 526.
[2025-11-13 06:24:34,315][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:24:34,315][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:24:54,109][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:25:04,737][__main__][INFO] - Number of regex retries in iteration 526: 1
[2025-11-13 06:25:04,738][__main__][INFO] - agents played in iteration 526 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:25:05,553][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:25:05,579][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:25:05,605][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:25:05,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:25:05,628][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:25:05,630][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:25:06,442][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:25:06,900][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:25:07,406][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:25:07,904][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:25:08,419][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:25:08,923][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:25:09,426][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:25:09,926][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:25:10,428][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:25:10,938][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:25:11,447][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:25:11,950][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:25:12,455][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:25:12,961][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:25:13,470][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:25:13,978][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:25:14,481][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:25:14,996][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:25:15,498][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:25:16,002][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:25:16,506][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:25:17,007][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:25:17,518][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:25:18,020][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:25:18,520][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:25:19,029][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:25:19,528][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:25:20,030][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:25:20,538][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:25:21,040][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:25:21,542][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:25:22,044][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:25:22,545][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:25:23,048][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:25:23,552][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:25:24,052][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:25:24,554][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:25:25,057][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:25:25,558][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:25:26,059][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:25:26,562][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:25:27,065][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:25:27,566][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:25:28,067][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:25:28,573][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:25:29,075][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:25:29,579][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:25:30,080][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:25:30,584][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:25:31,101][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:25:31,607][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:25:32,119][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:25:32,623][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:25:33,125][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:25:33,629][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:25:34,133][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:25:34,637][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:25:35,141][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:25:35,643][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:25:36,146][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:25:36,644][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:25:37,143][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:25:37,645][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:25:38,142][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:25:38,642][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10110 tokens.
[2025-11-13 06:25:39,456][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:33
[2025-11-13 06:25:40,202][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:25:40,204][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:25:40,205][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:25:41,212][__main__][INFO] - Iteration 527 took 1m 6s (45.48% Gen, 53.02% Train). Generation: 30s, Training: 35s. Estimated remaining time: 47h 28m 53s. Estimated total time: 55h 44m 53s. Time estimates for 10 more iterations: 11m 8s, 100 more iterations: 1h 51m 29s, 500 more iterations: 9h 17m 28s.
[2025-11-13 06:25:41,214][__main__][INFO] - Starting iteration 527.
[2025-11-13 06:25:41,688][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:25:41,688][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:26:08,332][__main__][INFO] - Number of regex retries in iteration 527: 0
[2025-11-13 06:26:08,334][__main__][INFO] - agents played in iteration 527 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:26:09,204][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:26:09,242][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:26:09,273][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:26:09,298][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:26:09,299][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:26:09,299][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:26:10,145][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:26:10,607][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:26:11,111][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:26:11,613][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:26:12,117][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:26:12,639][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:26:13,148][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:26:13,649][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:26:14,149][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:26:14,652][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:26:15,155][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:26:15,659][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:26:16,161][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:26:16,664][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:26:17,166][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:26:17,668][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:26:18,168][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:26:18,678][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:26:19,183][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:26:19,684][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:26:20,188][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:26:20,689][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:26:21,192][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:26:21,696][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:26:22,197][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:26:22,700][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:26:23,201][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:26:23,705][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:26:24,230][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:26:24,731][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:26:25,234][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:26:25,742][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:26:26,244][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:26:26,747][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:26:27,249][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:26:27,752][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:26:28,255][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:26:28,757][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:26:29,283][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:26:29,793][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:26:30,294][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:26:30,795][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:26:31,300][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:26:31,803][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:26:32,307][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:26:32,813][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:26:33,316][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:26:33,818][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:26:34,322][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:26:34,842][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:26:35,347][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:26:35,855][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:26:36,355][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:26:36,862][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:26:37,364][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:26:37,865][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:26:38,375][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:26:38,877][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:26:39,376][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:26:39,878][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:26:40,378][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:26:40,878][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:26:41,385][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:26:41,891][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:26:42,394][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10095 tokens.
[2025-11-13 06:26:43,315][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:33
[2025-11-13 06:26:43,981][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:26:43,983][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:26:43,985][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:26:44,953][__main__][INFO] - Iteration 528 took 1m 3s (42.12% Gen, 56.35% Train). Generation: 26s, Training: 35s. Estimated remaining time: 44h 26m 14s. Estimated total time: 52h 43m 17s. Time estimates for 10 more iterations: 10m 32s, 100 more iterations: 1h 45m 26s, 500 more iterations: 8h 47m 12s.
[2025-11-13 06:26:44,956][__main__][INFO] - Starting iteration 528.
[2025-11-13 06:26:45,440][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:26:45,440][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:27:08,272][__main__][INFO] - Number of regex retries in iteration 528: 0
[2025-11-13 06:27:08,273][__main__][INFO] - agents played in iteration 528 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:27:09,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:27:09,145][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:27:09,171][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:27:09,194][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:27:09,195][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:27:09,196][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:27:09,971][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:27:10,425][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:27:10,926][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:27:11,440][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:27:11,940][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:27:12,439][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:27:12,939][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:27:13,445][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:27:13,949][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:27:14,449][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:27:14,948][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:27:15,455][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:27:15,952][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:27:16,452][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:27:16,954][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:27:17,453][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:27:17,956][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:27:18,458][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:27:18,959][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:27:19,463][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:27:19,965][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:27:20,467][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:27:20,969][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:27:21,470][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:27:21,973][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:27:22,471][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:27:22,971][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:27:23,473][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:27:23,974][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:27:24,474][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:27:24,979][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:27:25,482][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:27:25,989][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:27:26,490][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:27:26,990][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:27:27,495][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:27:28,000][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:27:28,501][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:27:29,003][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:27:29,503][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:27:30,005][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:27:30,507][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:27:31,013][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:27:31,514][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:27:32,021][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:27:32,537][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:27:33,042][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:27:33,545][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:27:34,052][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:27:34,554][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:27:35,057][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:27:35,558][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:27:36,062][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:27:36,567][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:27:37,070][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:27:37,571][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:27:38,071][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:27:38,573][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:27:39,082][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:27:39,583][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:27:40,087][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:27:40,592][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:27:41,094][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:27:41,593][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:27:42,094][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9999 tokens.
[2025-11-13 06:27:42,985][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 62.08%, ΔTime: 00:00:33
[2025-11-13 06:27:43,704][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:27:43,705][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:27:43,707][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:27:46,529][__main__][INFO] - Iteration 529 took 1m 1s (37.37% Gen, 58.01% Train). Generation: 22s, Training: 35s. Estimated remaining time: 42h 36m 25s. Estimated total time: 50h 54m 30s. Time estimates for 10 more iterations: 10m 10s, 100 more iterations: 1h 41m 49s, 500 more iterations: 8h 29m 5s.
[2025-11-13 06:27:46,535][__main__][INFO] - Starting iteration 529.
[2025-11-13 06:27:47,028][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:27:47,028][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:28:09,171][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:28:20,889][__main__][INFO] - Number of regex retries in iteration 529: 1
[2025-11-13 06:28:20,890][__main__][INFO] - agents played in iteration 529 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:28:21,728][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:28:21,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:28:21,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:28:21,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:28:21,803][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:28:21,805][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:28:22,586][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:28:23,043][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:28:23,560][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:28:24,064][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:28:24,566][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:28:25,068][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:28:25,567][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:28:26,068][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:28:26,569][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:28:27,070][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:28:27,580][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:28:28,084][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:28:28,584][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:28:29,086][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:28:29,585][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:28:30,087][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:28:30,590][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:28:31,090][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:28:31,593][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:28:32,098][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:28:32,600][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:28:33,104][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:28:33,605][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:28:34,119][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:28:34,618][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:28:35,120][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:28:35,627][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:28:36,128][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:28:36,641][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:28:37,146][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:28:37,651][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:28:38,161][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:28:38,662][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:28:39,166][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:28:39,672][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:28:40,175][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:28:40,680][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:28:41,182][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:28:41,683][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:28:42,186][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:28:42,691][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:28:43,194][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:28:43,699][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:28:44,202][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:28:44,705][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:28:45,210][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:28:45,711][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:28:46,225][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:28:46,728][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:28:47,246][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:28:47,756][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:28:48,260][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:28:48,769][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:28:49,273][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:28:49,778][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:28:50,282][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:28:50,784][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:28:51,291][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:28:51,795][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:28:52,311][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:28:52,825][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:28:53,324][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:28:53,849][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:28:54,349][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:28:54,853][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10199 tokens.
[2025-11-13 06:28:55,767][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.45%, ΔTime: 00:00:33
[2025-11-13 06:28:56,422][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:28:56,425][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:28:56,428][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:28:57,471][__main__][INFO] - Iteration 530 took 1m 10s (48.07% Gen, 50.45% Train). Generation: 33s, Training: 35s. Estimated remaining time: 50h 22m 56s. Estimated total time: 58h 42m 12s. Time estimates for 10 more iterations: 11m 44s, 100 more iterations: 1h 57m 24s, 500 more iterations: 9h 47m 2s.
[2025-11-13 06:28:57,473][__main__][INFO] - Starting iteration 530.
[2025-11-13 06:28:57,959][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 06:28:57,960][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:29:24,185][__main__][INFO] - Number of regex retries in iteration 530: 0
[2025-11-13 06:29:24,186][__main__][INFO] - agents played in iteration 530 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:29:24,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:29:24,976][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:29:24,999][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:29:25,021][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:29:25,022][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:29:25,023][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:29:25,793][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:29:26,245][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:29:26,760][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:29:27,255][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:29:27,749][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:29:28,245][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:29:28,745][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:29:29,240][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:29:29,741][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:29:30,248][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:29:30,763][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:29:31,265][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:29:31,766][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:29:32,267][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:29:32,770][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:29:33,275][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:29:33,774][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:29:34,276][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:29:34,783][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:29:35,283][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:29:35,785][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:29:36,287][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:29:36,792][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:29:37,294][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:29:37,795][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:29:38,300][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:29:38,801][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:29:39,302][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:29:39,804][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:29:40,307][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:29:40,811][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:29:41,316][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:29:41,818][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:29:42,321][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:29:42,824][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:29:44,104][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:29:45,310][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:29:45,813][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:29:46,316][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:29:46,822][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:29:47,328][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:29:47,839][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:29:48,369][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:29:48,871][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:29:49,383][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:29:49,887][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:29:50,391][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:29:50,893][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:29:51,398][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:29:51,900][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:29:52,400][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:29:52,902][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:29:53,405][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:29:53,905][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:29:54,408][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:29:54,907][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:29:55,405][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:29:55,905][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:29:56,405][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:29:56,906][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:29:57,411][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:29:57,909][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:29:58,409][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:29:58,912][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:29:59,416][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10074 tokens.
[2025-11-13 06:30:00,302][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:34
[2025-11-13 06:30:00,971][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:30:00,973][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:30:00,975][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:30:02,870][__main__][INFO] - Iteration 531 took 1m 4s (40.40% Gen, 56.68% Train). Generation: 26s, Training: 36s. Estimated remaining time: 45h 45m 13s. Estimated total time: 54h 5m 34s. Time estimates for 10 more iterations: 10m 49s, 100 more iterations: 1h 48m 11s, 500 more iterations: 9h 0m 55s.
[2025-11-13 06:30:02,872][__main__][INFO] - Starting iteration 531.
[2025-11-13 06:30:03,353][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:30:03,354][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:30:21,711][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:30:25,336][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:30:27,121][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:30:30,582][__main__][INFO] - Number of regex retries in iteration 531: 3
[2025-11-13 06:30:30,583][__main__][INFO] - agents played in iteration 531 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:30:31,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:30:31,546][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:30:31,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:30:31,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:30:31,596][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:30:31,597][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:30:32,364][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:30:32,821][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:30:33,324][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:30:33,824][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:30:34,322][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:30:34,826][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:30:35,326][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:30:35,824][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:30:36,323][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:30:36,820][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:30:37,321][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:30:37,824][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:30:38,324][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:30:38,826][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:30:39,326][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:30:39,827][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:30:40,330][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:30:40,830][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:30:41,330][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:30:41,834][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:30:42,334][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:30:42,838][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:30:43,344][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:30:43,845][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:30:44,362][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:30:44,864][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:30:45,367][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:30:45,872][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:30:46,375][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:30:46,878][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:30:47,380][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:30:47,885][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:30:48,395][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:30:48,898][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:30:49,400][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:30:49,906][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:30:50,408][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:30:50,907][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:30:51,409][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:30:51,918][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:30:52,419][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:30:52,923][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:30:53,428][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:30:53,932][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:30:54,441][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:30:54,954][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:30:55,479][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:30:55,991][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:30:56,497][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:30:56,998][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:30:57,501][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:30:58,003][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:30:58,509][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:30:59,014][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:30:59,515][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:31:00,019][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:31:00,522][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:31:01,025][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:31:01,529][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:31:02,029][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:31:02,527][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:31:03,026][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:31:03,528][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:31:04,029][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:31:04,530][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10180 tokens.
[2025-11-13 06:31:05,411][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.43%, ΔTime: 00:00:33
[2025-11-13 06:31:06,152][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:31:06,155][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:31:06,157][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:31:07,232][__main__][INFO] - Iteration 532 took 1m 3s (42.62% Gen, 55.69% Train). Generation: 27s, Training: 35s. Estimated remaining time: 44h 52m 31s. Estimated total time: 53h 13m 57s. Time estimates for 10 more iterations: 10m 38s, 100 more iterations: 1h 46m 27s, 500 more iterations: 8h 52m 19s.
[2025-11-13 06:31:07,234][__main__][INFO] - Starting iteration 532.
[2025-11-13 06:31:07,710][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:31:07,711][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:31:26,543][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:31:26,632][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:31:38,633][__main__][INFO] - Number of regex retries in iteration 532: 2
[2025-11-13 06:31:38,633][__main__][INFO] - agents played in iteration 532 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:31:39,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:31:39,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:31:39,634][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:31:39,665][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:31:39,666][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:31:39,666][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:31:40,450][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:31:40,911][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:31:41,412][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:31:41,908][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:31:42,403][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:31:42,899][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:31:43,403][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:31:43,901][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:31:44,396][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:31:44,901][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:31:45,406][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:31:45,908][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:31:46,408][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:31:46,906][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:31:47,412][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:31:47,915][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:31:48,421][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:31:48,919][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:31:49,419][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:31:49,921][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:31:50,423][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:31:50,923][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:31:51,427][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:31:51,927][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:31:52,429][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:31:52,930][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:31:53,437][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:31:53,945][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:31:54,450][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:31:54,952][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:31:55,455][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:31:55,958][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:31:56,464][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:31:56,969][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:31:57,476][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:31:57,980][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:31:58,482][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:31:58,984][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:31:59,485][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:31:59,987][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:32:00,502][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:32:01,005][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:32:01,508][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:32:02,014][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:32:02,516][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:32:03,022][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:32:03,524][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:32:04,028][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:32:04,537][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:32:05,038][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:32:05,544][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:32:06,046][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:32:06,547][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:32:07,053][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:32:07,555][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:32:08,055][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:32:08,556][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:32:09,057][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:32:09,563][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:32:10,065][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:32:10,566][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:32:11,068][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:32:11,568][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:32:12,073][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:32:12,575][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10081 tokens.
[2025-11-13 06:32:13,469][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.44%, ΔTime: 00:00:33
[2025-11-13 06:32:14,124][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:32:14,127][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:32:14,129][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:32:15,038][__main__][INFO] - Iteration 533 took 1m 7s (45.93% Gen, 52.72% Train). Generation: 30s, Training: 35s. Estimated remaining time: 47h 43m 51s. Estimated total time: 56h 6m 25s. Time estimates for 10 more iterations: 11m 13s, 100 more iterations: 1h 52m 12s, 500 more iterations: 9h 21m 4s.
[2025-11-13 06:32:15,040][__main__][INFO] - Starting iteration 533.
[2025-11-13 06:32:15,519][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:32:15,520][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:32:37,233][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:32:49,077][__main__][INFO] - Number of regex retries in iteration 533: 1
[2025-11-13 06:32:49,077][__main__][INFO] - agents played in iteration 533 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:32:49,916][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:32:49,939][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:32:49,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:32:49,983][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:32:49,984][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:32:49,985][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:32:50,756][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:32:51,207][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:32:51,725][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:32:52,223][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:32:52,720][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:32:53,219][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:32:53,717][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:32:54,231][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:32:54,732][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:32:55,235][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:32:55,739][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:32:56,240][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:32:56,746][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:32:57,248][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:32:57,749][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:32:58,259][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:32:58,760][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:32:59,265][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:32:59,769][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:33:00,275][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:33:00,784][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:33:01,291][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:33:01,796][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:33:02,309][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:33:02,813][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:33:03,327][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:33:03,830][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:33:04,333][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:33:04,848][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:33:05,350][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:33:05,857][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:33:06,360][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:33:06,865][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:33:07,371][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:33:07,875][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:33:08,380][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:33:08,892][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:33:09,398][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:33:09,902][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:33:10,406][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:33:10,911][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:33:11,416][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:33:11,922][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:33:12,423][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:33:12,925][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:33:13,431][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:33:13,945][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:33:14,446][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:33:14,953][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:33:15,452][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:33:15,953][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:33:16,454][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:33:16,954][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:33:17,453][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:33:17,959][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:33:18,462][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:33:18,965][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:33:19,464][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:33:19,969][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:33:20,474][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:33:20,976][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:33:21,481][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:33:21,982][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:33:22,485][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:33:22,986][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10192 tokens.
[2025-11-13 06:33:23,835][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:33
[2025-11-13 06:33:24,582][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:33:24,584][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:33:24,585][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:33:25,633][__main__][INFO] - Iteration 534 took 1m 10s (47.86% Gen, 50.64% Train). Generation: 33s, Training: 35s. Estimated remaining time: 50h 2m 1s. Estimated total time: 58h 25m 45s. Time estimates for 10 more iterations: 11m 41s, 100 more iterations: 1h 56m 51s, 500 more iterations: 9h 44m 17s.
[2025-11-13 06:33:25,635][__main__][INFO] - Starting iteration 534.
[2025-11-13 06:33:26,105][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:33:26,105][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:33:59,260][__main__][INFO] - Number of regex retries in iteration 534: 0
[2025-11-13 06:33:59,262][__main__][INFO] - agents played in iteration 534 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:34:00,150][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:34:00,199][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:34:00,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:34:00,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:34:00,270][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:34:00,271][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:34:01,175][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:34:01,635][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:34:02,147][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:34:02,648][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:34:03,151][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:34:03,668][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:34:04,174][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:34:04,675][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:34:05,173][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:34:05,685][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:34:06,191][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:34:06,699][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:34:07,200][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:34:07,705][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:34:08,207][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:34:08,714][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:34:09,219][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:34:09,721][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:34:10,234][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:34:10,741][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:34:11,252][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:34:11,755][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:34:12,257][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:34:12,761][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:34:13,263][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:34:13,765][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:34:14,266][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:34:14,771][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:34:15,300][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:34:15,807][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:34:16,308][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:34:16,815][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:34:17,318][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:34:17,820][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:34:18,321][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:34:18,821][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:34:19,328][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:34:19,828][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:34:20,328][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:34:20,831][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:34:21,335][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:34:21,834][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:34:22,333][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:34:22,831][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:34:23,332][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:34:23,831][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:34:24,335][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:34:24,833][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:34:25,330][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:34:25,828][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:34:26,324][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:34:26,824][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:34:27,330][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:34:27,829][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:34:28,344][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:34:28,844][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:34:29,348][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:34:29,851][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:34:30,353][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:34:30,853][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:34:31,351][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:34:31,850][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:34:32,364][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:34:32,866][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:34:33,372][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10079 tokens.
[2025-11-13 06:34:34,215][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:33
[2025-11-13 06:34:34,877][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:34:34,879][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:34:34,881][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:34:35,795][__main__][INFO] - Iteration 535 took 1m 9s (47.58% Gen, 51.11% Train). Generation: 33s, Training: 35s. Estimated remaining time: 49h 39m 37s. Estimated total time: 58h 4m 31s. Time estimates for 10 more iterations: 11m 36s, 100 more iterations: 1h 56m 9s, 500 more iterations: 9h 40m 45s.
[2025-11-13 06:34:35,797][__main__][INFO] - Starting iteration 535.
[2025-11-13 06:34:36,280][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:34:36,281][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:34:50,665][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:35:02,107][__main__][INFO] - Number of regex retries in iteration 535: 1
[2025-11-13 06:35:02,107][__main__][INFO] - agents played in iteration 535 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:35:02,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:35:03,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:35:03,030][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:35:03,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:35:03,053][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:35:03,054][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:35:03,900][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:35:04,356][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:35:04,875][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:35:05,376][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:35:05,875][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:35:06,379][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:35:06,882][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:35:07,386][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:35:07,888][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:35:08,390][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:35:08,895][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:35:09,401][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:35:09,901][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:35:10,405][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:35:10,906][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:35:11,407][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:35:11,908][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:35:12,410][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:35:12,913][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:35:13,417][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:35:13,920][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:35:14,423][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:35:14,925][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:35:15,427][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:35:15,944][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:35:16,446][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:35:16,948][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:35:17,450][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:35:17,953][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:35:18,454][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:35:20,644][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:35:21,158][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:35:21,661][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:35:22,163][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:35:22,674][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:35:23,174][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:35:23,674][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:35:24,175][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:35:24,677][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:35:25,184][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:35:25,688][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:35:26,191][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:35:26,694][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:35:27,193][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:35:27,697][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:35:28,198][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:35:28,701][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:35:29,204][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:35:29,727][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:35:30,228][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:35:30,736][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:35:31,236][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:35:31,742][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:35:32,242][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:35:32,748][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:35:33,263][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:35:33,765][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:35:34,264][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:35:34,763][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:35:35,265][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:35:35,773][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:35:36,271][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:35:36,774][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:35:37,285][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:35:37,784][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10058 tokens.
[2025-11-13 06:35:38,648][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:34
[2025-11-13 06:35:39,313][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:35:39,314][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:35:39,316][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:35:40,362][__main__][INFO] - Iteration 536 took 1m 4s (40.30% Gen, 58.06% Train). Generation: 25s, Training: 37s. Estimated remaining time: 44h 58m 7s. Estimated total time: 53h 24m 6s. Time estimates for 10 more iterations: 10m 40s, 100 more iterations: 1h 46m 48s, 500 more iterations: 8h 54m 1s.
[2025-11-13 06:35:40,364][__main__][INFO] - Starting iteration 536.
[2025-11-13 06:35:40,863][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:35:40,863][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:36:16,269][__main__][INFO] - Number of regex retries in iteration 536: 0
[2025-11-13 06:36:16,270][__main__][INFO] - agents played in iteration 536 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:36:17,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:36:17,146][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:36:17,171][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:36:17,194][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:36:17,194][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:36:17,195][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:36:18,086][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:36:18,552][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:36:19,060][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:36:19,585][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:36:20,089][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:36:20,591][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:36:21,096][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:36:21,603][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:36:22,105][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:36:22,607][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:36:23,109][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:36:23,612][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:36:24,115][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:36:24,615][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:36:25,122][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:36:25,622][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:36:26,124][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:36:26,626][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:36:27,129][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:36:27,643][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:36:28,146][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:36:28,657][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:36:29,162][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:36:29,665][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:36:30,183][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:36:30,686][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:36:31,193][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:36:31,697][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:36:32,197][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:36:32,701][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:36:33,204][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:36:33,703][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:36:34,212][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:36:34,712][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:36:35,210][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:36:35,710][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:36:36,208][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:36:36,709][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:36:37,211][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:36:37,714][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:36:38,215][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:36:38,715][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:36:39,219][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:36:39,724][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:36:40,227][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:36:40,736][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:36:41,234][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:36:41,734][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:36:42,249][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:36:42,750][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:36:43,254][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:36:43,754][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:36:44,252][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:36:44,754][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:36:45,254][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:36:45,755][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:36:46,259][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:36:46,756][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:36:47,275][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:36:47,777][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:36:48,278][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:36:48,787][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:36:49,291][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:36:49,788][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:36:50,287][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10213 tokens.
[2025-11-13 06:36:51,089][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 06:36:51,877][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:36:51,878][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:36:51,880][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:36:52,945][__main__][INFO] - Iteration 537 took 1m 12s (49.12% Gen, 49.40% Train). Generation: 35s, Training: 35s. Estimated remaining time: 51h 36m 57s. Estimated total time: 60h 4m 8s. Time estimates for 10 more iterations: 12m 0s, 100 more iterations: 2h 0m 8s, 500 more iterations: 10h 0m 41s.
[2025-11-13 06:36:52,947][__main__][INFO] - Starting iteration 537.
[2025-11-13 06:36:53,455][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:36:53,456][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:37:15,309][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 20 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:37:20,519][__main__][INFO] - Number of regex retries in iteration 537: 1
[2025-11-13 06:37:20,519][__main__][INFO] - agents played in iteration 537 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:37:21,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:37:21,411][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:37:21,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:37:21,466][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:37:21,467][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:37:21,468][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:37:22,350][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:37:22,814][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:37:23,319][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:37:23,824][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:37:24,328][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:37:24,829][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:37:25,333][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:37:25,843][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:37:26,346][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:37:26,860][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:37:27,362][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:37:27,865][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:37:28,370][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:37:28,873][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:37:30,493][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:37:31,023][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:37:31,526][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:37:32,030][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:37:32,534][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:37:33,040][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:37:33,555][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:37:34,059][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:37:34,563][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:37:35,069][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:37:35,572][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:37:36,080][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:37:36,581][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:37:37,085][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:37:37,591][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:37:38,092][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:37:38,594][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:37:39,100][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:37:39,600][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:37:40,104][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:37:40,607][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:37:41,110][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:37:41,610][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:37:42,111][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:37:42,619][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:37:43,127][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:37:43,628][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:37:44,135][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:37:44,635][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:37:45,139][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:37:45,642][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:37:46,142][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:37:46,663][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:37:47,167][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:37:47,667][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:37:48,172][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:37:48,670][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:37:49,175][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:37:49,675][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:37:50,171][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:37:50,685][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:37:51,184][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:37:51,685][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:37:52,192][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:37:52,695][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:37:53,198][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:37:53,699][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:37:54,199][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:37:54,705][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:37:55,204][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:37:55,703][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10167 tokens.
[2025-11-13 06:37:56,551][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:34
[2025-11-13 06:37:57,207][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:37:57,209][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:37:57,210][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:37:58,171][__main__][INFO] - Iteration 538 took 1m 4s (41.82% Gen, 56.70% Train). Generation: 27s, Training: 36s. Estimated remaining time: 45h 27m 31s. Estimated total time: 53h 55m 47s. Time estimates for 10 more iterations: 10m 47s, 100 more iterations: 1h 47m 51s, 500 more iterations: 8h 59m 17s.
[2025-11-13 06:37:58,173][__main__][INFO] - Starting iteration 538.
[2025-11-13 06:37:58,659][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:37:58,659][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:38:37,933][__main__][INFO] - Number of regex retries in iteration 538: 0
[2025-11-13 06:38:37,934][__main__][INFO] - agents played in iteration 538 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:38:38,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:38:38,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:38:38,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:38:38,868][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:38:38,869][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:38:38,869][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:38:39,664][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:38:40,122][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:38:40,628][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:38:41,127][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:38:41,636][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:38:42,144][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:38:42,646][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:38:43,150][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:38:43,660][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:38:44,164][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:38:44,680][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:38:45,184][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:38:45,686][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:38:46,201][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:38:46,716][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:38:47,229][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:38:47,733][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:38:48,232][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:38:48,734][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:38:49,237][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:38:49,742][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:38:50,244][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:38:50,748][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:38:51,251][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:38:51,764][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:38:52,267][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:38:52,770][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:38:53,273][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:38:53,772][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:38:54,281][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:38:54,780][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:38:55,286][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:38:55,795][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:38:56,299][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:38:56,803][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:38:57,304][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:38:57,805][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:38:58,310][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:38:58,814][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:38:59,313][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:38:59,816][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:39:00,317][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:39:00,815][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:39:01,314][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:39:01,812][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:39:02,311][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:39:02,811][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:39:03,309][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:39:03,809][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:39:04,310][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:39:04,812][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:39:05,310][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:39:05,810][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:39:06,312][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:39:06,814][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:39:07,312][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:39:07,811][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:39:08,313][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:39:08,815][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:39:09,315][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:39:09,816][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:39:10,316][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:39:10,819][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:39:11,321][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:39:11,820][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10092 tokens.
[2025-11-13 06:39:12,673][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:33
[2025-11-13 06:39:13,425][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:39:13,426][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:39:13,428][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:39:14,504][__main__][INFO] - Iteration 539 took 1m 15s (51.78% Gen, 46.80% Train). Generation: 39s, Training: 35s. Estimated remaining time: 54h 42m 46s. Estimated total time: 63h 12m 19s. Time estimates for 10 more iterations: 12m 38s, 100 more iterations: 2h 6m 24s, 500 more iterations: 10h 32m 3s.
[2025-11-13 06:39:14,506][__main__][INFO] - Starting iteration 539.
[2025-11-13 06:39:14,996][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:39:14,997][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:39:30,439][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:39:41,659][__main__][INFO] - Number of regex retries in iteration 539: 1
[2025-11-13 06:39:41,660][__main__][INFO] - agents played in iteration 539 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:39:42,536][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:39:42,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:39:42,594][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:39:42,618][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:39:42,619][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:39:42,619][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:39:43,513][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:39:43,972][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:39:44,495][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:39:44,998][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:39:45,497][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:39:46,000][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:39:46,500][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:39:47,004][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:39:47,503][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:39:48,004][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:39:48,508][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:39:49,014][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:39:49,518][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:39:50,020][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:39:50,521][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:39:51,025][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:39:51,526][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:39:52,026][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:39:52,526][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:39:53,027][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:39:53,531][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:39:54,031][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:39:54,530][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:39:55,031][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:39:55,534][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:39:56,070][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:39:56,573][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:39:57,077][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:39:57,590][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:39:58,090][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:39:58,594][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:39:59,096][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:39:59,603][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:40:00,104][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:40:00,608][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:40:01,107][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:40:01,614][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:40:02,120][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:40:02,625][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:40:03,125][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:40:03,625][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:40:04,127][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:40:04,627][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:40:05,130][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:40:05,632][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:40:06,132][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:40:06,632][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:40:07,137][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:40:07,643][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:40:08,144][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:40:08,647][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:40:09,150][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:40:09,649][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:40:10,150][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:40:10,648][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:40:11,150][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:40:11,650][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:40:12,149][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:40:12,655][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:40:13,152][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:40:13,653][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:40:14,150][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:40:14,655][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:40:15,154][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:40:15,652][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10052 tokens.
[2025-11-13 06:40:16,476][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:32
[2025-11-13 06:40:17,101][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:40:17,102][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:40:17,104][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:40:18,042][__main__][INFO] - Iteration 540 took 1m 3s (42.29% Gen, 56.22% Train). Generation: 26s, Training: 35s. Estimated remaining time: 44h 1m 43s. Estimated total time: 52h 32m 19s. Time estimates for 10 more iterations: 10m 30s, 100 more iterations: 1h 45m 4s, 500 more iterations: 8h 45m 23s.
[2025-11-13 06:40:18,044][__main__][INFO] - Starting iteration 540.
[2025-11-13 06:40:18,551][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 06:40:18,552][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:40:46,598][__main__][INFO] - Number of regex retries in iteration 540: 0
[2025-11-13 06:40:46,599][__main__][INFO] - agents played in iteration 540 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:40:47,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:40:47,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:40:47,526][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:40:47,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:40:47,551][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:40:47,551][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:40:48,427][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:40:48,886][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:40:49,394][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:40:49,896][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:40:50,404][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:40:50,904][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:40:51,403][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:40:51,908][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:40:52,408][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:40:52,906][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:40:53,407][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:40:53,904][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:40:54,406][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:40:54,905][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:40:55,403][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:40:55,993][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:40:56,490][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:40:56,989][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:40:57,488][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:40:57,988][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:40:58,501][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:40:59,002][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:40:59,501][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:41:00,008][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:41:00,508][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:41:01,018][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:41:01,518][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:41:02,020][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:41:02,528][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:41:03,032][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:41:03,537][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:41:04,037][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:41:04,540][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:41:05,046][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:41:05,550][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:41:06,052][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:41:06,558][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:41:07,060][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:41:07,566][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:41:08,069][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:41:08,572][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:41:09,085][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:41:09,608][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:41:10,119][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:41:10,624][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:41:11,126][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:41:11,630][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:41:12,133][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:41:12,635][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:41:13,136][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:41:13,636][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:41:14,140][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:41:14,641][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:41:15,145][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:41:15,648][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:41:16,153][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:41:16,650][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:41:17,151][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:41:17,652][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:41:18,153][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:41:18,652][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:41:19,151][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:41:19,653][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:41:20,150][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:41:20,650][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10069 tokens.
[2025-11-13 06:41:21,470][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 06:41:22,212][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:41:22,213][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:41:22,215][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:41:24,177][__main__][INFO] - Iteration 541 took 1m 5s (42.74% Gen, 54.27% Train). Generation: 28s, Training: 35s. Estimated remaining time: 46h 9m 36s. Estimated total time: 54h 41m 18s. Time estimates for 10 more iterations: 10m 56s, 100 more iterations: 1h 49m 22s, 500 more iterations: 9h 6m 53s.
[2025-11-13 06:41:24,179][__main__][INFO] - Starting iteration 541.
[2025-11-13 06:41:24,655][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:41:24,655][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:41:56,801][__main__][INFO] - Number of regex retries in iteration 541: 0
[2025-11-13 06:41:56,803][__main__][INFO] - agents played in iteration 541 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:41:57,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:41:57,680][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:41:57,705][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:41:57,728][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:41:57,729][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:41:57,729][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:41:58,591][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:41:59,047][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:41:59,554][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:42:00,055][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:42:00,565][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:42:01,066][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:42:01,565][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:42:02,064][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:42:02,570][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:42:03,068][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:42:03,566][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:42:04,068][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:42:04,566][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:42:05,066][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:42:05,564][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:42:06,062][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:42:06,567][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:42:07,075][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:42:07,579][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:42:08,095][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:42:08,597][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:42:09,114][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:42:09,617][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:42:10,121][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:42:10,632][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:42:11,138][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:42:11,639][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:42:12,146][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:42:12,647][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:42:13,152][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:42:13,652][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:42:14,156][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:42:14,660][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:42:15,164][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:42:15,664][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:42:16,165][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:42:16,666][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:42:17,169][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:42:17,669][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:42:18,173][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:42:18,672][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:42:19,177][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:42:19,687][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:42:20,191][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:42:20,691][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:42:21,204][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:42:21,708][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:42:22,220][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:42:22,723][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:42:23,226][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:42:23,732][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:42:24,230][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:42:24,734][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:42:25,239][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:42:25,744][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:42:26,247][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:42:26,749][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:42:27,252][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:42:27,751][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:42:28,256][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:42:28,756][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:42:29,257][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:42:29,756][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:42:30,257][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:42:30,755][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10194 tokens.
[2025-11-13 06:42:31,661][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.43%, ΔTime: 00:00:33
[2025-11-13 06:42:32,307][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:42:32,310][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:42:32,312][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:42:33,290][__main__][INFO] - Iteration 542 took 1m 8s (46.84% Gen, 51.74% Train). Generation: 32s, Training: 35s. Estimated remaining time: 48h 38m 56s. Estimated total time: 57h 11m 48s. Time estimates for 10 more iterations: 11m 26s, 100 more iterations: 1h 54m 23s, 500 more iterations: 9h 31m 58s.
[2025-11-13 06:42:33,293][__main__][INFO] - Starting iteration 542.
[2025-11-13 06:42:33,789][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:42:33,790][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:43:03,301][__main__][INFO] - Number of regex retries in iteration 542: 0
[2025-11-13 06:43:03,305][__main__][INFO] - agents played in iteration 542 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:43:04,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:43:04,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:43:04,235][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:43:04,264][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:43:04,264][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:43:04,265][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:43:05,090][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:43:05,545][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:43:06,050][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:43:06,552][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:43:07,051][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:43:07,550][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:43:08,052][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:43:08,554][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:43:09,055][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:43:09,560][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:43:10,061][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:43:10,562][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:43:11,071][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:43:11,572][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:43:12,077][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:43:12,580][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:43:13,082][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:43:13,636][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:43:14,139][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:43:14,638][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:43:15,144][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:43:15,650][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:43:16,151][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:43:16,653][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:43:17,152][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:43:17,652][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:43:18,149][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:43:18,645][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:43:19,141][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:43:19,641][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:43:20,141][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:43:20,638][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:43:21,142][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:43:21,644][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:43:22,143][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:43:22,640][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:43:23,134][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:43:23,633][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:43:24,134][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:43:24,630][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:43:25,127][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:43:25,627][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:43:26,125][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:43:26,630][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:43:27,128][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:43:27,628][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:43:28,137][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:43:28,637][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:43:29,137][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:43:29,633][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:43:30,143][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:43:30,653][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:43:31,163][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:43:31,672][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:43:32,181][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:43:32,680][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:43:33,182][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:43:33,681][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:43:34,181][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:43:34,680][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:43:35,184][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:43:35,684][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:43:36,186][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:43:36,687][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:43:37,186][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10050 tokens.
[2025-11-13 06:43:38,082][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:33
[2025-11-13 06:43:38,758][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:43:38,761][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:43:38,763][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:43:39,670][__main__][INFO] - Iteration 543 took 1m 5s (44.80% Gen, 53.82% Train). Generation: 29s, Training: 35s. Estimated remaining time: 46h 20m 7s. Estimated total time: 54h 54m 5s. Time estimates for 10 more iterations: 10m 58s, 100 more iterations: 1h 49m 48s, 500 more iterations: 9h 9m 0s.
[2025-11-13 06:43:39,672][__main__][INFO] - Starting iteration 543.
[2025-11-13 06:43:40,170][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:43:40,171][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:43:59,575][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:44:01,508][mllm.models.large_language_model_local][WARNING] - Response Proposal: 10 hats, 0 books, 0 balls

Given the per-item values, hats are worth much less to me compared to books and balls, while Alice seems to value hats much higher than the other items. By proposing to take all the hats, I aim to secure a significant portion of my total points, as the low value of hats to Alice will not significantly affect her decision to accept or the proportional allocation if exceeded. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:44:09,154][__main__][INFO] - Number of regex retries in iteration 543: 2
[2025-11-13 06:44:09,154][__main__][INFO] - agents played in iteration 543 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:44:09,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:44:09,970][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:44:10,000][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:44:10,025][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:44:10,026][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:44:10,027][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:44:10,859][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:44:11,316][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:44:11,823][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:44:12,323][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:44:12,844][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:44:13,349][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:44:13,849][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:44:14,353][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:44:14,856][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:44:15,364][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:44:15,868][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:44:16,372][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:44:16,878][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:44:17,382][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:44:17,882][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:44:18,383][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:44:18,883][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:44:19,385][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:44:19,885][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:44:20,389][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:44:20,898][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:44:21,401][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:44:21,906][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:44:22,406][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:44:22,907][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:44:23,429][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:44:23,927][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:44:24,424][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:44:24,923][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:44:25,421][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:44:25,934][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:44:26,430][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:44:26,928][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:44:27,441][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:44:27,943][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:44:28,445][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:44:28,943][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:44:29,443][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:44:29,949][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:44:30,448][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:44:30,947][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:44:31,442][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:44:31,938][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:44:32,433][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:44:32,928][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:44:33,425][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:44:33,933][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:44:34,427][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:44:34,922][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:44:35,445][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:44:35,942][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:44:36,439][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:44:36,936][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:44:37,432][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:44:37,938][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:44:38,433][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:44:38,928][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:44:39,443][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:44:39,940][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:44:40,437][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:44:40,935][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:44:41,431][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:44:41,941][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:44:42,441][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:44:42,936][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10040 tokens.
[2025-11-13 06:44:43,757][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:32
[2025-11-13 06:44:44,514][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:44:44,515][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:44:44,517][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:44:45,515][__main__][INFO] - Iteration 544 took 1m 5s (44.35% Gen, 54.12% Train). Generation: 28s, Training: 35s. Estimated remaining time: 45h 52m 13s. Estimated total time: 54h 27m 17s. Time estimates for 10 more iterations: 10m 53s, 100 more iterations: 1h 48m 54s, 500 more iterations: 9h 4m 32s.
[2025-11-13 06:44:45,517][__main__][INFO] - Starting iteration 544.
[2025-11-13 06:44:45,998][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:44:45,999][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:44:58,071][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:45:08,266][__main__][INFO] - Number of regex retries in iteration 544: 1
[2025-11-13 06:45:08,270][__main__][INFO] - agents played in iteration 544 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:45:09,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:45:09,196][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:45:09,223][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:45:09,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:45:09,248][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:45:09,249][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:45:10,096][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:45:10,635][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:45:11,152][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:45:11,657][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:45:12,164][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:45:12,666][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:45:13,167][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:45:13,673][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:45:14,172][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:45:14,673][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:45:15,178][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:45:15,678][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:45:16,181][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:45:16,679][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:45:17,183][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:45:17,684][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:45:18,185][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:45:18,689][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:45:19,193][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:45:19,695][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:45:20,200][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:45:20,704][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:45:21,208][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:45:21,707][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:45:22,208][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:45:22,706][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:45:23,209][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:45:23,711][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:45:24,225][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:45:24,726][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:45:25,233][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:45:25,730][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:45:26,233][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:45:26,734][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:45:27,235][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:45:27,740][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:45:28,243][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:45:28,744][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:45:29,253][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:45:29,752][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:45:30,252][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:45:30,755][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:45:31,254][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:45:31,755][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:45:32,254][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:45:32,755][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:45:33,266][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:45:33,764][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:45:34,266][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:45:34,770][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:45:35,288][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:45:35,793][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:45:36,294][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:45:36,877][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:45:37,391][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:45:37,892][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:45:38,393][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:45:38,892][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:45:39,393][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:45:39,890][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:45:40,387][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:45:40,885][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:45:41,389][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:45:41,886][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:45:42,385][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10073 tokens.
[2025-11-13 06:45:43,302][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:33
[2025-11-13 06:45:43,991][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:45:43,992][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:45:43,994][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:45:45,031][__main__][INFO] - Iteration 545 took 59s (37.73% Gen, 60.51% Train). Generation: 22s, Training: 35s. Estimated remaining time: 40h 35m 39s. Estimated total time: 49h 11m 42s. Time estimates for 10 more iterations: 9m 50s, 100 more iterations: 1h 38m 23s, 500 more iterations: 8h 11m 57s.
[2025-11-13 06:45:45,033][__main__][INFO] - Starting iteration 545.
[2025-11-13 06:45:45,518][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:45:45,518][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:46:15,122][__main__][INFO] - Number of regex retries in iteration 545: 0
[2025-11-13 06:46:15,122][__main__][INFO] - agents played in iteration 545 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:46:15,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:46:16,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:46:16,026][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:46:16,048][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:46:16,049][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:46:16,050][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:46:16,908][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:46:17,361][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:46:17,872][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:46:18,381][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:46:18,884][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:46:19,390][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:46:19,893][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:46:20,402][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:46:20,906][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:46:21,410][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:46:21,914][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:46:22,414][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:46:22,915][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:46:23,423][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:46:23,927][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:46:24,427][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:46:24,928][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:46:25,434][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:46:25,953][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:46:26,457][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:46:26,962][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:46:27,466][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:46:27,968][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:46:28,476][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:46:28,978][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:46:29,480][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:46:29,983][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:46:30,490][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:46:30,996][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:46:31,500][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:46:32,008][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:46:32,511][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:46:33,016][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:46:33,516][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:46:34,027][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:46:34,527][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:46:35,035][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:46:35,537][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:46:36,041][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:46:36,557][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:46:37,064][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:46:37,564][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:46:38,068][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:46:38,572][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:46:39,076][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:46:39,575][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:46:40,080][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:46:40,581][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:46:41,084][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:46:41,582][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:46:42,080][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:46:42,576][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:46:43,082][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:46:43,581][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:46:44,081][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:46:44,578][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:46:45,074][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:46:45,577][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:46:46,075][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:46:46,578][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:46:47,081][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:46:47,581][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:46:48,084][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:46:48,586][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:46:49,086][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10084 tokens.
[2025-11-13 06:46:49,928][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:33
[2025-11-13 06:46:50,710][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:46:50,711][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:46:50,713][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:46:51,803][__main__][INFO] - Iteration 546 took 1m 6s (44.66% Gen, 53.69% Train). Generation: 29s, Training: 35s. Estimated remaining time: 46h 37m 6s. Estimated total time: 55h 14m 16s. Time estimates for 10 more iterations: 11m 2s, 100 more iterations: 1h 50m 28s, 500 more iterations: 9h 12m 22s.
[2025-11-13 06:46:51,805][__main__][INFO] - Starting iteration 546.
[2025-11-13 06:46:52,283][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:46:52,284][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:47:16,038][__main__][INFO] - Number of regex retries in iteration 546: 0
[2025-11-13 06:47:16,039][__main__][INFO] - agents played in iteration 546 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:47:16,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:47:16,861][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:47:16,886][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:47:16,908][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:47:16,909][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:47:16,910][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:47:17,724][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:47:18,183][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:47:18,693][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:47:19,196][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:47:19,697][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:47:20,196][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:47:20,695][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:47:21,199][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:47:21,699][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:47:22,199][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:47:22,706][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:47:23,203][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:47:23,705][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:47:24,206][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:47:24,705][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:47:25,204][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:47:25,702][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:47:26,201][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:47:26,696][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:47:27,192][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:47:27,695][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:47:28,190][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:47:28,685][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:47:29,194][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:47:29,691][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:47:30,193][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:47:30,688][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:47:31,188][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:47:31,693][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:47:32,188][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:47:32,685][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:47:33,187][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:47:33,688][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:47:34,190][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:47:34,689][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:47:35,186][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:47:35,696][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:47:36,191][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:47:36,689][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:47:37,188][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:47:37,688][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:47:38,192][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:47:38,692][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:47:39,190][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:47:39,688][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:47:40,186][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:47:40,686][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:47:41,188][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:47:41,686][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:47:42,192][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:47:42,696][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:47:43,195][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:47:43,691][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:47:44,190][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:47:44,691][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:47:45,195][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:47:45,695][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:47:46,194][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:47:47,239][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:47:48,315][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:47:48,818][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:47:49,317][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:47:49,815][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:47:50,312][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:47:50,815][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10092 tokens.
[2025-11-13 06:47:51,775][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:34
[2025-11-13 06:47:52,468][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:47:52,470][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:47:52,472][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:47:53,419][__main__][INFO] - Iteration 547 took 1m 1s (38.85% Gen, 59.59% Train). Generation: 23s, Training: 36s. Estimated remaining time: 42h 18m 37s. Estimated total time: 50h 56m 49s. Time estimates for 10 more iterations: 10m 11s, 100 more iterations: 1h 41m 53s, 500 more iterations: 8h 29m 28s.
[2025-11-13 06:47:53,421][__main__][INFO] - Starting iteration 547.
[2025-11-13 06:47:53,918][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:47:53,918][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:48:12,331][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:48:24,129][__main__][INFO] - Number of regex retries in iteration 547: 1
[2025-11-13 06:48:24,129][__main__][INFO] - agents played in iteration 547 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:48:24,926][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:48:24,952][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:48:24,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:48:24,999][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:48:25,000][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:48:25,001][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:48:25,793][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:48:26,246][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:48:26,752][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:48:27,250][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:48:27,752][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:48:28,250][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:48:28,751][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:48:29,248][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:48:29,752][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:48:30,254][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:48:30,765][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:48:31,270][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:48:31,771][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:48:32,275][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:48:32,776][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:48:33,291][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:48:33,814][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:48:34,320][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:48:34,820][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:48:35,319][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:48:35,818][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:48:36,317][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:48:36,817][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:48:37,322][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:48:37,823][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:48:38,325][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:48:38,831][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:48:39,328][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:48:39,829][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:48:40,327][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:48:40,823][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:48:41,322][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:48:41,825][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:48:42,325][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:48:42,827][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:48:43,330][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:48:43,833][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:48:44,330][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:48:44,831][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:48:45,327][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:48:45,826][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:48:46,324][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:48:46,823][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:48:47,321][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:48:47,823][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:48:48,323][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:48:48,820][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:48:49,320][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:48:49,818][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:48:50,317][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:48:50,818][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:48:51,314][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:48:51,812][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:48:52,310][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:48:52,810][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:48:53,308][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:48:53,808][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:48:54,312][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:48:54,813][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:48:55,310][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:48:55,807][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:48:56,304][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:48:56,810][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:48:57,313][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:48:57,813][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10164 tokens.
[2025-11-13 06:48:58,700][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:32
[2025-11-13 06:48:59,484][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:48:59,485][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:48:59,487][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:49:00,590][__main__][INFO] - Iteration 548 took 1m 6s (45.31% Gen, 53.03% Train). Generation: 30s, Training: 35s. Estimated remaining time: 46h 54m 20s. Estimated total time: 55h 33m 39s. Time estimates for 10 more iterations: 11m 6s, 100 more iterations: 1h 51m 7s, 500 more iterations: 9h 15m 36s.
[2025-11-13 06:49:00,592][__main__][INFO] - Starting iteration 548.
[2025-11-13 06:49:01,113][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:49:01,114][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:49:12,672][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:49:18,590][__main__][INFO] - Number of regex retries in iteration 548: 1
[2025-11-13 06:49:18,591][__main__][INFO] - agents played in iteration 548 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:49:19,483][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:49:19,507][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:49:19,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:49:19,553][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:49:19,554][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:49:19,555][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:49:20,457][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:49:21,821][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:49:22,343][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:49:22,844][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:49:23,348][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:49:23,855][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:49:24,360][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:49:24,875][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:49:25,376][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:49:25,898][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:49:26,402][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:49:26,904][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:49:27,405][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:49:27,909][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:49:28,416][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:49:28,916][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:49:29,416][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:49:29,921][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:49:30,424][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:49:30,927][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:49:31,428][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:49:31,930][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:49:32,434][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:49:32,936][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:49:33,448][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:49:33,949][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:49:34,449][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:49:34,952][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:49:35,453][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:49:35,954][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:49:36,462][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:49:36,962][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:49:37,462][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:49:37,976][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:49:38,473][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:49:38,998][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:49:39,500][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:49:40,002][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:49:40,498][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:49:40,999][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:49:41,505][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:49:42,006][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:49:42,505][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:49:43,005][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:49:43,501][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:49:44,000][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:49:44,497][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:49:44,997][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:49:45,510][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:49:46,009][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:49:46,508][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:49:47,010][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:49:47,508][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:49:48,012][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:49:48,509][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:49:49,008][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:49:49,508][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:49:50,007][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:49:50,508][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:49:51,009][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:49:51,507][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:49:52,018][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:49:52,519][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:49:53,019][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:49:53,520][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10051 tokens.
[2025-11-13 06:49:54,464][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:34
[2025-11-13 06:49:55,099][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:49:55,101][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:49:55,102][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:49:56,341][__main__][INFO] - Iteration 549 took 55s (31.64% Gen, 66.11% Train). Generation: 17s, Training: 36s. Estimated remaining time: 37h 21m 11s. Estimated total time: 46h 1m 26s. Time estimates for 10 more iterations: 9m 12s, 100 more iterations: 1h 32m 2s, 500 more iterations: 7h 40m 14s.
[2025-11-13 06:49:56,344][__main__][INFO] - Starting iteration 549.
[2025-11-13 06:49:56,828][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:49:56,829][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:50:25,290][__main__][INFO] - Number of regex retries in iteration 549: 0
[2025-11-13 06:50:25,291][__main__][INFO] - agents played in iteration 549 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:50:26,146][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:50:26,174][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:50:26,201][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:50:26,224][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:50:26,225][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:50:26,226][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:50:27,092][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:50:27,561][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:50:28,064][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:50:28,568][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:50:29,068][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:50:29,567][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:50:30,066][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:50:30,564][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:50:31,065][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:50:31,567][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:50:32,066][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:50:32,563][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:50:33,064][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:50:33,562][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:50:34,063][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:50:34,561][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:50:35,058][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:50:35,563][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:50:36,061][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:50:36,564][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:50:37,061][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:50:37,562][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:50:38,062][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:50:38,564][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:50:39,064][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:50:39,561][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:50:40,061][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:50:40,561][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:50:41,058][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:50:41,557][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:50:42,059][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:50:42,557][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:50:43,057][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:50:43,558][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:50:44,060][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:50:44,559][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:50:45,057][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:50:45,557][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:50:46,057][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:50:46,555][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:50:47,057][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:50:47,554][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:50:48,052][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:50:48,553][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:50:49,050][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:50:49,550][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:50:50,051][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:50:50,550][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:50:51,048][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:50:51,547][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:50:52,044][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:50:52,544][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:50:53,043][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:50:53,540][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:50:54,043][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:50:54,540][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:50:55,036][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:50:55,534][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:50:56,031][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:50:56,532][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:50:57,032][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:50:57,532][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:50:58,031][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:50:58,532][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:50:59,033][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10047 tokens.
[2025-11-13 06:50:59,921][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:32
[2025-11-13 06:51:00,704][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:51:00,706][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:51:00,708][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:51:01,726][__main__][INFO] - Iteration 550 took 1m 4s (43.86% Gen, 54.57% Train). Generation: 28s, Training: 35s. Estimated remaining time: 45h 23m 35s. Estimated total time: 54h 4m 55s. Time estimates for 10 more iterations: 10m 48s, 100 more iterations: 1h 48m 9s, 500 more iterations: 9h 0m 49s.
[2025-11-13 06:51:01,728][__main__][INFO] - Starting iteration 550.
[2025-11-13 06:51:02,212][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 06:51:02,213][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:51:14,275][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 06:51:21,626][__main__][INFO] - Number of regex retries in iteration 550: 1
[2025-11-13 06:51:21,628][__main__][INFO] - agents played in iteration 550 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:51:22,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:51:22,679][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:51:22,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:51:22,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:51:22,737][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:51:22,738][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:51:23,666][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:51:24,216][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:51:24,738][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:51:25,246][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:51:25,752][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:51:26,256][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:51:26,769][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:51:27,277][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:51:27,780][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:51:28,284][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:51:28,785][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:51:29,286][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:51:29,793][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:51:30,296][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:51:30,819][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:51:31,319][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:51:31,828][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:51:32,332][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:51:32,832][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:51:33,334][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:51:33,838][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:51:34,339][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:51:34,850][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:51:35,349][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:51:35,850][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:51:36,350][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:51:36,849][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:51:37,354][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:51:37,853][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:51:38,354][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:51:38,853][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:51:39,354][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:51:39,853][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:51:40,352][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:51:40,852][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:51:41,359][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:51:41,859][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:51:42,360][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:51:42,860][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:51:43,360][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:51:43,872][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:51:44,376][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:51:44,873][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:51:45,373][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:51:45,871][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:51:46,367][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:51:46,868][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:51:47,376][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:51:47,880][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:51:48,379][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:51:48,879][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:51:49,383][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:51:49,881][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:51:50,385][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:51:50,885][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:51:51,385][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:51:51,887][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:51:52,394][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:51:52,896][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:51:53,409][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:51:53,910][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:51:54,414][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:51:54,915][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:51:55,412][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:51:55,927][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10131 tokens.
[2025-11-13 06:51:56,745][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:33
[2025-11-13 06:51:57,421][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:51:57,423][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:51:57,424][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:51:59,371][__main__][INFO] - Iteration 551 took 57s (33.96% Gen, 62.62% Train). Generation: 19s, Training: 35s. Estimated remaining time: 38h 55m 42s. Estimated total time: 47h 38m 0s. Time estimates for 10 more iterations: 9m 31s, 100 more iterations: 1h 35m 16s, 500 more iterations: 7h 56m 20s.
[2025-11-13 06:51:59,374][__main__][INFO] - Starting iteration 551.
[2025-11-13 06:51:59,871][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 06:51:59,871][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:52:29,686][__main__][INFO] - Number of regex retries in iteration 551: 0
[2025-11-13 06:52:29,687][__main__][INFO] - agents played in iteration 551 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:52:30,528][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:52:30,553][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:52:30,576][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:52:30,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:52:30,600][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:52:30,600][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:52:31,515][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:52:31,977][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:52:32,485][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:52:32,996][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:52:33,503][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:52:34,008][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:52:34,513][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:52:35,021][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:52:35,526][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:52:36,029][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:52:36,527][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:52:37,030][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:52:37,529][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:52:38,045][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:52:38,545][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:52:39,046][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:52:39,557][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:52:40,060][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:52:40,566][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:52:41,069][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:52:41,571][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:52:42,077][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:52:42,580][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:52:43,084][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:52:43,589][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:52:44,093][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:52:44,595][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:52:45,097][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:52:45,600][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:52:46,106][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:52:46,609][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:52:47,109][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:52:47,612][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:52:48,113][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:52:48,613][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:52:49,115][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:52:49,618][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:52:50,125][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:52:50,629][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:52:51,135][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:52:51,638][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:52:52,141][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:52:52,657][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:52:53,158][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:52:53,660][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:52:54,162][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:52:54,662][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:52:55,173][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:52:55,674][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:52:56,174][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:52:56,673][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:52:57,176][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:52:57,680][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:52:58,178][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:52:58,680][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:52:59,181][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:52:59,683][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:53:00,185][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:53:00,684][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:53:01,183][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:53:01,686][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:53:02,186][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:53:02,687][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:53:03,188][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:53:03,687][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10103 tokens.
[2025-11-13 06:53:04,590][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:33
[2025-11-13 06:53:05,351][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:53:05,352][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:53:05,354][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:53:06,430][__main__][INFO] - Iteration 552 took 1m 6s (44.80% Gen, 53.59% Train). Generation: 29s, Training: 35s. Estimated remaining time: 46h 44m 35s. Estimated total time: 55h 28m 0s. Time estimates for 10 more iterations: 11m 5s, 100 more iterations: 1h 50m 56s, 500 more iterations: 9h 14m 40s.
[2025-11-13 06:53:06,432][__main__][INFO] - Starting iteration 552.
[2025-11-13 06:53:06,903][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 06:53:06,904][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:53:30,113][__main__][INFO] - Number of regex retries in iteration 552: 0
[2025-11-13 06:53:30,113][__main__][INFO] - agents played in iteration 552 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:53:30,918][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:53:30,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:53:30,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:53:30,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:53:30,989][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:53:30,989][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:53:31,840][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:53:32,296][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:53:32,804][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:53:33,306][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:53:33,813][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:53:34,313][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:53:34,813][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:53:35,311][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:53:35,875][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:53:36,381][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:53:36,881][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:53:37,387][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:53:37,889][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:53:38,391][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:53:38,892][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:53:39,392][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:53:39,895][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:53:40,393][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:53:40,892][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:53:41,389][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:53:41,888][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:53:42,389][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:53:42,890][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:53:43,389][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:53:43,888][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:53:44,385][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:53:44,887][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:53:45,388][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:53:45,889][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:53:46,391][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:53:46,888][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:53:47,384][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:53:47,881][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:53:48,376][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:53:48,872][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:53:49,367][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:53:49,864][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:53:50,372][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:53:50,867][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:53:51,363][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:53:51,870][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:53:52,366][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:53:52,864][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:53:53,362][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:53:53,862][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:53:54,368][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:53:54,866][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:53:55,366][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:53:55,865][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:53:56,363][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:53:56,863][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:53:57,363][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:53:57,863][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:53:59,186][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:53:59,849][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:54:00,348][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:54:00,849][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:54:01,351][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:54:01,853][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:54:02,369][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:54:02,870][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:54:03,373][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:54:03,875][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:54:04,375][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:54:04,879][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10073 tokens.
[2025-11-13 06:54:05,820][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:33
[2025-11-13 06:54:06,497][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:54:06,499][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:54:06,501][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:54:07,524][__main__][INFO] - Iteration 553 took 1m 0s (38.29% Gen, 60.03% Train). Generation: 23s, Training: 36s. Estimated remaining time: 41h 46m 39s. Estimated total time: 50h 31m 5s. Time estimates for 10 more iterations: 10m 6s, 100 more iterations: 1h 41m 2s, 500 more iterations: 8h 25m 10s.
[2025-11-13 06:54:07,527][__main__][INFO] - Starting iteration 553.
[2025-11-13 06:54:08,023][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 06:54:08,024][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:54:32,357][__main__][INFO] - Number of regex retries in iteration 553: 0
[2025-11-13 06:54:32,357][__main__][INFO] - agents played in iteration 553 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:54:33,354][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:54:33,388][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:54:33,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:54:33,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:54:33,435][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:54:33,436][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:54:34,316][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:54:34,782][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:54:35,291][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:54:35,797][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:54:36,305][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:54:36,810][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:54:37,316][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:54:37,824][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:54:38,328][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:54:38,829][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:54:39,329][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:54:39,829][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:54:40,339][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:54:40,840][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:54:41,343][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:54:41,844][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:54:42,344][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:54:42,862][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:54:43,364][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:54:43,867][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:54:44,369][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:54:44,868][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:54:45,380][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:54:45,883][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:54:46,386][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:54:46,890][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:54:47,390][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:54:47,889][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:54:48,392][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:54:48,894][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:54:49,395][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:54:49,896][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:54:50,398][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:54:50,898][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:54:51,404][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:54:51,904][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:54:52,406][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:54:52,909][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:54:53,410][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:54:53,909][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:54:54,409][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:54:54,912][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:54:55,415][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:54:55,915][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:54:56,411][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:54:56,909][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:54:57,409][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:54:57,907][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:54:58,406][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:54:58,904][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:54:59,402][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:54:59,899][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:55:00,395][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:55:00,891][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:55:01,391][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:55:01,890][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:55:02,388][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:55:02,889][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:55:03,390][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:55:03,889][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:55:04,385][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:55:04,883][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:55:05,388][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:55:05,885][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:55:06,383][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10056 tokens.
[2025-11-13 06:55:07,232][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 06:55:07,995][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:55:07,996][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:55:07,999][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:55:09,007][__main__][INFO] - Iteration 554 took 1m 0s (39.90% Gen, 58.44% Train). Generation: 24s, Training: 35s. Estimated remaining time: 42h 3m 46s. Estimated total time: 50h 49m 13s. Time estimates for 10 more iterations: 10m 9s, 100 more iterations: 1h 41m 38s, 500 more iterations: 8h 28m 12s.
[2025-11-13 06:55:09,010][__main__][INFO] - Starting iteration 554.
[2025-11-13 06:55:09,511][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 06:55:09,511][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:55:36,769][__main__][INFO] - Number of regex retries in iteration 554: 0
[2025-11-13 06:55:36,770][__main__][INFO] - agents played in iteration 554 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:55:37,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:55:37,592][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:55:37,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:55:37,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:55:37,641][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:55:37,641][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:55:38,510][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:55:38,970][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:55:39,478][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:55:39,984][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:55:40,490][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:55:40,996][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:55:41,502][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:55:42,008][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:55:42,515][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:55:43,016][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:55:43,521][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:55:44,022][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:55:44,524][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:55:45,026][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:55:45,527][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:55:46,028][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:55:46,530][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:55:47,031][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:55:47,532][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:55:48,031][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:55:48,530][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:55:49,029][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:55:49,535][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:55:50,038][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:55:50,541][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:55:51,046][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:55:51,553][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:55:52,061][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:55:52,565][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:55:53,071][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:55:53,576][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:55:54,085][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:55:56,084][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:55:56,598][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:55:57,101][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:55:57,604][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:55:58,111][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:55:58,616][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:55:59,118][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:55:59,619][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:56:00,124][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:56:00,626][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:56:01,129][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:56:01,637][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:56:02,142][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:56:02,643][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:56:03,148][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:56:03,657][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:56:04,159][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:56:04,673][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:56:05,175][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:56:05,677][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:56:06,197][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:56:06,703][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:56:07,216][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:56:07,726][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:56:08,228][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:56:08,736][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:56:09,244][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:56:09,746][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:56:10,252][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:56:10,754][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:56:11,255][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:56:11,758][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:56:12,265][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10085 tokens.
[2025-11-13 06:56:13,166][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:34
[2025-11-13 06:56:13,823][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:56:13,825][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:56:13,827][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:56:14,823][__main__][INFO] - Iteration 555 took 1m 5s (41.73% Gen, 56.74% Train). Generation: 27s, Training: 37s. Estimated remaining time: 45h 39m 6s. Estimated total time: 54h 25m 39s. Time estimates for 10 more iterations: 10m 53s, 100 more iterations: 1h 48m 51s, 500 more iterations: 9h 4m 16s.
[2025-11-13 06:56:14,826][__main__][INFO] - Starting iteration 555.
[2025-11-13 06:56:15,335][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 06:56:15,335][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:56:49,011][__main__][INFO] - Number of regex retries in iteration 555: 0
[2025-11-13 06:56:49,012][__main__][INFO] - agents played in iteration 555 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:56:49,854][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:56:49,877][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:56:49,900][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:56:49,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:56:49,923][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:56:49,924][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:56:50,797][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:56:51,255][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:56:51,765][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:56:52,276][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:56:52,781][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:56:53,283][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:56:53,790][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:56:54,290][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:56:54,804][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:56:55,305][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:56:55,808][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:56:56,309][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:56:56,811][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:56:57,310][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:56:57,807][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:56:58,307][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:56:58,817][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:56:59,318][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:56:59,817][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:57:00,318][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:57:00,824][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:57:01,327][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:57:01,831][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:57:02,328][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:57:02,833][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:57:03,329][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:57:03,830][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:57:04,328][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:57:04,830][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:57:05,334][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:57:05,832][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:57:06,333][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:57:06,839][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:57:07,342][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:57:07,848][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:57:08,373][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:57:08,874][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:57:09,394][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:57:09,898][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:57:10,401][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:57:10,902][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:57:11,399][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:57:11,905][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:57:12,404][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:57:12,902][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:57:13,405][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:57:13,904][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:57:14,407][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:57:14,908][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:57:15,404][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:57:15,913][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:57:16,410][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:57:16,910][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:57:17,410][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:57:17,907][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:57:18,405][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:57:18,906][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:57:19,402][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:57:19,910][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:57:20,408][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:57:20,906][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:57:21,407][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:57:21,907][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:57:22,412][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:57:22,913][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10195 tokens.
[2025-11-13 06:57:23,812][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:33
[2025-11-13 06:57:24,574][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:57:24,576][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:57:24,577][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:57:25,612][__main__][INFO] - Iteration 556 took 1m 10s (47.92% Gen, 50.61% Train). Generation: 33s, Training: 35s. Estimated remaining time: 49h 46m 11s. Estimated total time: 58h 33m 55s. Time estimates for 10 more iterations: 11m 42s, 100 more iterations: 1h 57m 7s, 500 more iterations: 9h 45m 39s.
[2025-11-13 06:57:25,615][__main__][INFO] - Starting iteration 556.
[2025-11-13 06:57:26,084][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 06:57:26,084][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:57:50,810][__main__][INFO] - Number of regex retries in iteration 556: 0
[2025-11-13 06:57:50,812][__main__][INFO] - agents played in iteration 556 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:57:51,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:57:51,694][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:57:51,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:57:51,741][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:57:51,741][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:57:51,742][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:57:52,669][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:57:53,128][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:57:53,636][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:57:54,135][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:57:54,635][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:57:55,137][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:57:55,637][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:57:56,137][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:57:56,638][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:57:57,144][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:57:57,653][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:57:58,159][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:57:58,661][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:57:59,168][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:57:59,671][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:58:00,174][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:58:00,679][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:58:01,182][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:58:01,687][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:58:02,192][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:58:02,700][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:58:03,206][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:58:03,712][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:58:04,226][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:58:04,730][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:58:05,235][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:58:05,738][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:58:06,241][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:58:06,744][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:58:07,243][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:58:07,745][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:58:08,251][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:58:08,755][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:58:09,254][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:58:09,757][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:58:10,254][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:58:10,761][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:58:11,262][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:58:11,761][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:58:12,264][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:58:12,766][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:58:13,267][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:58:13,768][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:58:14,267][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:58:14,768][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:58:15,270][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:58:15,776][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:58:16,275][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:58:16,780][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:58:17,291][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:58:17,790][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:58:18,292][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:58:18,812][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:58:19,314][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:58:19,818][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:58:20,319][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:58:20,818][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:58:21,334][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:58:21,834][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:58:22,337][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:58:22,836][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:58:23,335][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:58:23,842][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:58:24,344][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:58:24,845][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10095 tokens.
[2025-11-13 06:58:25,654][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 06:58:26,309][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:58:26,310][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:58:26,312][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:58:27,206][__main__][INFO] - Iteration 557 took 1m 1s (40.46% Gen, 58.08% Train). Generation: 24s, Training: 35s. Estimated remaining time: 42h 7m 24s. Estimated total time: 50h 56m 10s. Time estimates for 10 more iterations: 10m 11s, 100 more iterations: 1h 41m 52s, 500 more iterations: 8h 29m 21s.
[2025-11-13 06:58:27,208][__main__][INFO] - Starting iteration 557.
[2025-11-13 06:58:27,704][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 06:58:27,704][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:59:00,083][__main__][INFO] - Number of regex retries in iteration 557: 0
[2025-11-13 06:59:00,084][__main__][INFO] - agents played in iteration 557 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 06:59:00,937][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:59:00,973][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:59:01,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:59:01,025][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 06:59:01,026][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 06:59:01,027][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 06:59:01,908][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 06:59:02,368][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 06:59:02,875][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 06:59:03,380][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 06:59:03,884][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 06:59:04,387][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 06:59:04,890][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 06:59:05,392][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 06:59:05,895][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 06:59:06,416][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 06:59:06,924][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 06:59:07,425][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 06:59:07,931][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 06:59:08,436][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 06:59:08,939][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 06:59:09,443][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 06:59:09,946][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 06:59:10,454][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 06:59:10,956][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 06:59:11,462][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 06:59:11,967][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 06:59:12,473][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 06:59:12,977][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 06:59:13,486][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 06:59:14,010][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 06:59:14,516][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 06:59:15,014][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 06:59:15,525][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 06:59:16,030][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 06:59:16,532][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 06:59:17,032][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 06:59:17,531][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 06:59:18,035][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 06:59:18,535][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 06:59:19,037][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 06:59:19,535][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 06:59:20,033][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 06:59:20,532][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 06:59:21,045][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 06:59:21,545][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 06:59:22,047][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 06:59:22,549][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 06:59:23,054][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 06:59:23,559][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 06:59:24,065][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 06:59:24,569][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 06:59:25,074][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 06:59:25,577][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 06:59:26,085][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 06:59:26,583][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 06:59:27,086][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 06:59:27,585][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 06:59:28,086][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 06:59:28,591][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 06:59:29,097][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 06:59:29,595][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 06:59:30,095][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 06:59:30,593][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 06:59:31,091][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 06:59:31,602][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 06:59:32,102][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 06:59:32,603][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 06:59:33,105][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 06:59:33,606][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 06:59:34,124][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10090 tokens.
[2025-11-13 06:59:34,981][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 06:59:35,724][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 06:59:35,726][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 06:59:35,727][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 06:59:36,797][__main__][INFO] - Iteration 558 took 1m 9s (46.86% Gen, 51.59% Train). Generation: 32s, Training: 35s. Estimated remaining time: 48h 44m 47s. Estimated total time: 57h 34m 42s. Time estimates for 10 more iterations: 11m 30s, 100 more iterations: 1h 55m 9s, 500 more iterations: 9h 35m 47s.
[2025-11-13 06:59:36,799][__main__][INFO] - Starting iteration 558.
[2025-11-13 06:59:37,266][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 06:59:37,267][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 06:59:55,011][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:00:05,757][__main__][INFO] - Number of regex retries in iteration 558: 1
[2025-11-13 07:00:05,758][__main__][INFO] - agents played in iteration 558 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:00:06,630][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:00:06,665][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:00:06,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:00:06,715][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:00:06,716][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:00:06,717][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:00:07,599][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:00:08,061][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:00:08,566][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:00:09,066][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:00:09,568][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:00:10,068][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:00:10,568][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:00:11,071][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:00:11,571][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:00:12,075][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:00:12,586][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:00:13,090][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:00:13,597][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:00:14,105][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:00:14,605][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:00:15,106][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:00:15,608][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:00:16,114][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:00:16,622][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:00:17,125][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:00:17,631][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:00:18,133][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:00:18,644][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:00:19,147][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:00:19,656][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:00:20,165][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:00:20,669][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:00:21,172][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:00:21,684][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:00:22,196][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:00:22,706][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:00:23,211][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:00:23,720][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:00:24,230][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:00:24,734][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:00:25,248][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:00:25,754][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:00:26,257][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:00:26,769][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:00:27,276][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:00:27,781][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:00:28,288][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:00:28,794][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:00:29,305][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:00:29,808][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:00:30,310][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:00:30,825][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:00:31,330][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:00:31,833][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:00:32,338][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:00:32,848][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:00:33,356][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:00:33,858][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:00:34,366][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:00:34,873][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:00:35,376][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:00:35,878][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:00:36,380][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:00:36,886][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:00:37,389][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:00:37,898][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:00:38,402][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:00:38,906][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:00:39,412][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:00:39,913][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10145 tokens.
[2025-11-13 07:00:40,804][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:33
[2025-11-13 07:00:41,471][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:00:41,473][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:00:41,476][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:00:42,422][__main__][INFO] - Iteration 559 took 1m 5s (43.73% Gen, 54.82% Train). Generation: 28s, Training: 35s. Estimated remaining time: 45h 26m 46s. Estimated total time: 54h 17m 46s. Time estimates for 10 more iterations: 10m 51s, 100 more iterations: 1h 48m 35s, 500 more iterations: 9h 2m 57s.
[2025-11-13 07:00:42,425][__main__][INFO] - Starting iteration 559.
[2025-11-13 07:00:42,917][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 07:00:42,918][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:01:03,150][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:01:13,591][__main__][INFO] - Number of regex retries in iteration 559: 1
[2025-11-13 07:01:13,592][__main__][INFO] - agents played in iteration 559 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:01:14,493][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:01:14,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:01:14,544][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:01:14,567][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:01:14,567][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:01:14,568][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:01:15,355][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:01:15,824][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:01:16,329][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:01:16,833][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:01:17,342][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:01:17,841][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:01:18,359][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:01:18,858][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:01:19,359][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:01:19,867][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:01:20,370][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:01:20,877][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:01:21,381][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:01:21,884][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:01:22,389][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:01:22,893][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:01:23,397][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:01:23,897][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:01:24,399][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:01:24,903][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:01:25,407][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:01:25,908][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:01:26,408][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:01:26,910][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:01:27,413][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:01:27,917][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:01:28,423][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:01:28,929][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:01:29,431][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:01:29,933][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:01:30,435][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:01:30,938][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:01:31,455][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:01:31,961][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:01:32,465][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:01:32,969][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:01:33,472][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:01:33,984][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:01:34,486][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:01:34,990][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:01:35,489][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:01:35,987][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:01:36,487][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:01:36,987][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:01:37,489][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:01:37,990][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:01:38,491][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:01:38,989][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:01:39,493][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:01:39,994][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:01:40,491][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:01:40,993][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:01:41,493][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:01:41,993][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:01:42,488][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:01:42,988][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:01:43,489][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:01:43,986][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:01:44,488][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:01:44,984][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:01:45,482][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:01:45,986][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:01:46,484][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:01:46,985][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:01:47,488][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10113 tokens.
[2025-11-13 07:01:48,315][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 07:01:49,069][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:01:49,070][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:01:49,072][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:01:50,112][__main__][INFO] - Iteration 560 took 1m 7s (45.65% Gen, 52.80% Train). Generation: 30s, Training: 35s. Estimated remaining time: 47h 7m 36s. Estimated total time: 55h 59m 45s. Time estimates for 10 more iterations: 11m 11s, 100 more iterations: 1h 51m 59s, 500 more iterations: 9h 19m 57s.
[2025-11-13 07:01:50,114][__main__][INFO] - Starting iteration 560.
[2025-11-13 07:01:50,581][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 07:01:50,581][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:02:17,820][__main__][INFO] - Number of regex retries in iteration 560: 0
[2025-11-13 07:02:17,823][__main__][INFO] - agents played in iteration 560 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:02:18,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:02:18,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:02:18,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:02:18,796][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:02:18,796][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:02:18,797][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:02:19,713][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:02:20,184][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:02:20,688][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:02:21,195][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:02:21,696][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:02:22,196][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:02:22,703][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:02:23,205][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:02:23,708][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:02:24,208][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:02:24,707][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:02:25,208][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:02:25,710][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:02:26,209][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:02:26,712][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:02:27,212][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:02:27,714][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:02:28,218][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:02:28,727][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:02:29,228][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:02:29,729][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:02:30,231][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:02:30,735][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:02:31,241][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:02:31,746][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:02:32,251][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:02:32,755][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:02:33,264][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:02:33,775][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:02:34,281][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:02:34,783][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:02:35,289][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:02:35,803][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:02:36,305][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:02:36,807][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:02:37,314][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:02:37,818][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:02:38,324][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:02:38,829][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:02:39,331][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:02:39,840][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:02:40,345][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:02:40,849][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:02:41,365][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:02:41,872][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:02:42,377][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:02:42,908][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:02:43,412][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:02:43,920][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:02:44,424][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:02:44,927][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:02:45,430][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:02:45,934][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:02:46,446][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:02:46,949][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:02:47,452][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:02:47,962][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:02:48,463][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:02:48,976][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:02:49,478][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:02:49,980][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:02:50,491][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:02:50,993][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:02:51,498][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:02:52,007][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10128 tokens.
[2025-11-13 07:02:52,873][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:33
[2025-11-13 07:02:53,519][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:02:53,521][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:02:53,523][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:02:55,377][__main__][INFO] - Iteration 561 took 1m 4s (42.04% Gen, 55.09% Train). Generation: 27s, Training: 35s. Estimated remaining time: 45h 6m 38s. Estimated total time: 53h 59m 51s. Time estimates for 10 more iterations: 10m 47s, 100 more iterations: 1h 47m 59s, 500 more iterations: 8h 59m 58s.
[2025-11-13 07:02:55,380][__main__][INFO] - Starting iteration 561.
[2025-11-13 07:02:55,980][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:02:55,981][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:03:27,878][__main__][INFO] - Number of regex retries in iteration 561: 0
[2025-11-13 07:03:27,878][__main__][INFO] - agents played in iteration 561 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:03:28,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:03:28,691][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:03:28,717][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:03:28,741][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:03:28,741][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:03:28,742][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:03:29,536][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:03:29,990][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:03:30,492][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:03:30,996][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:03:31,496][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:03:31,999][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:03:32,506][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:03:33,008][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:03:33,514][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:03:34,027][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:03:34,528][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:03:35,034][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:03:35,537][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:03:36,037][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:03:36,536][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:03:37,037][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:03:37,541][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:03:38,054][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:03:38,560][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:03:39,066][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:03:39,577][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:03:40,080][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:03:40,603][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:03:41,108][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:03:41,610][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:03:42,113][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:03:42,614][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:03:43,120][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:03:43,624][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:03:44,127][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:03:44,632][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:03:45,131][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:03:45,631][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:03:46,131][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:03:47,371][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:03:48,522][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:03:49,031][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:03:49,531][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:03:50,032][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:03:50,536][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:03:51,040][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:03:51,539][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:03:52,047][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:03:52,552][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:03:53,055][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:03:53,571][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:03:54,073][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:03:54,573][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:03:55,079][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:03:55,581][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:03:56,091][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:03:56,591][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:03:57,091][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:03:57,600][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:03:58,105][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:03:58,607][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:03:59,109][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:03:59,612][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:04:00,116][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:04:00,623][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:04:01,128][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:04:01,634][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:04:02,136][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:04:02,641][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:04:03,148][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10150 tokens.
[2025-11-13 07:04:04,119][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.22%, Current % of VRAM taken: 60.48%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:34
[2025-11-13 07:04:04,766][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:04:04,767][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:04:04,769][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:04:05,764][__main__][INFO] - Iteration 562 took 1m 9s (45.71% Gen, 52.86% Train). Generation: 31s, Training: 36s. Estimated remaining time: 49h 14m 47s. Estimated total time: 58h 9m 11s. Time estimates for 10 more iterations: 11m 37s, 100 more iterations: 1h 56m 18s, 500 more iterations: 9h 41m 31s.
[2025-11-13 07:04:05,767][__main__][INFO] - Starting iteration 562.
[2025-11-13 07:04:06,246][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:04:06,246][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:04:30,019][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:04:37,100][__main__][INFO] - Number of regex retries in iteration 562: 1
[2025-11-13 07:04:37,101][__main__][INFO] - agents played in iteration 562 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:04:37,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:04:37,986][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:04:38,011][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:04:38,036][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:04:38,037][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:04:38,037][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:04:38,894][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:04:39,360][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:04:39,865][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:04:40,371][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:04:40,877][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:04:41,376][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:04:41,875][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:04:42,379][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:04:42,881][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:04:43,402][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:04:43,906][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:04:44,408][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:04:44,910][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:04:45,411][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:04:45,919][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:04:46,428][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:04:46,929][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:04:47,429][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:04:47,930][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:04:48,440][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:04:48,976][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:04:49,479][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:04:49,990][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:04:50,498][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:04:51,002][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:04:51,509][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:04:52,013][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:04:52,521][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:04:53,031][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:04:53,534][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:04:54,043][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:04:54,545][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:04:55,049][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:04:55,548][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:04:56,046][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:04:56,545][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:04:57,047][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:04:57,545][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:04:58,046][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:04:58,544][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:04:59,045][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:04:59,545][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:05:00,043][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:05:00,541][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:05:01,039][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:05:01,535][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:05:02,034][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:05:02,532][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:05:03,029][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:05:03,529][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:05:04,028][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:05:04,530][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:05:05,027][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:05:05,526][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:05:06,029][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:05:06,528][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:05:07,028][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:05:07,526][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:05:08,025][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:05:08,524][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:05:09,022][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:05:09,522][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:05:10,022][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:05:10,524][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:05:11,023][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10136 tokens.
[2025-11-13 07:05:11,828][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 07:05:12,754][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:05:12,755][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:05:12,769][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:05:13,985][__main__][INFO] - Iteration 563 took 1m 7s (45.55% Gen, 52.66% Train). Generation: 30s, Training: 35s. Estimated remaining time: 47h 31m 26s. Estimated total time: 56h 26m 58s. Time estimates for 10 more iterations: 11m 17s, 100 more iterations: 1h 52m 53s, 500 more iterations: 9h 24m 29s.
[2025-11-13 07:05:13,986][__main__][INFO] - Starting iteration 563.
[2025-11-13 07:05:14,475][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:05:14,476][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:05:28,827][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:05:29,082][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:05:39,962][__main__][INFO] - Number of regex retries in iteration 563: 2
[2025-11-13 07:05:39,963][__main__][INFO] - agents played in iteration 563 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:05:40,791][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:05:40,817][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:05:40,842][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:05:40,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:05:40,865][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:05:40,865][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:05:41,756][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:05:42,217][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:05:42,727][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:05:44,291][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:05:45,470][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:05:45,971][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:05:46,471][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:05:46,971][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:05:47,472][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:05:47,973][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:05:48,476][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:05:48,979][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:05:49,480][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:05:49,983][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:05:50,497][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:05:51,006][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:05:51,511][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:05:52,017][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:05:52,521][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:05:53,031][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:05:53,533][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:05:54,033][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:05:54,536][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:05:55,043][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:05:55,544][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:05:56,045][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:05:56,547][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:05:57,051][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:05:57,552][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:05:58,053][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:05:58,554][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:05:59,059][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:05:59,565][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:06:00,071][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:06:00,573][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:06:01,087][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:06:01,592][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:06:02,107][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:06:02,609][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:06:03,116][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:06:03,629][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:06:04,135][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:06:04,641][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:06:05,148][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:06:05,650][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:06:06,153][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:06:06,653][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:06:07,150][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:06:07,650][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:06:08,150][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:06:08,653][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:06:09,152][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:06:09,652][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:06:10,153][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:06:10,655][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:06:11,155][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:06:11,654][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:06:12,157][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:06:12,661][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:06:13,161][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:06:13,662][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:06:14,167][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:06:14,669][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:06:15,174][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:06:15,673][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10164 tokens.
[2025-11-13 07:06:16,594][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.11%, ΔTime: 00:00:34
[2025-11-13 07:06:17,236][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:06:17,237][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:06:17,239][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:06:18,159][__main__][INFO] - Iteration 564 took 1m 3s (40.02% Gen, 58.53% Train). Generation: 25s, Training: 37s. Estimated remaining time: 44h 7m 37s. Estimated total time: 53h 4m 14s. Time estimates for 10 more iterations: 10m 36s, 100 more iterations: 1h 46m 8s, 500 more iterations: 8h 50m 42s.
[2025-11-13 07:06:18,161][__main__][INFO] - Starting iteration 564.
[2025-11-13 07:06:18,652][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:06:18,653][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:06:48,764][__main__][INFO] - Number of regex retries in iteration 564: 0
[2025-11-13 07:06:48,765][__main__][INFO] - agents played in iteration 564 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:06:49,561][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:06:49,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:06:49,609][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:06:49,632][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:06:49,632][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:06:49,633][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:06:50,468][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:06:50,933][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:06:51,439][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:06:51,952][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:06:52,452][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:06:52,957][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:06:53,465][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:06:53,973][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:06:54,475][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:06:54,979][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:06:55,487][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:06:55,991][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:06:56,496][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:06:57,000][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:06:57,530][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:06:58,043][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:06:58,549][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:06:59,054][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:06:59,560][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:07:00,069][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:07:00,572][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:07:01,079][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:07:01,583][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:07:02,087][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:07:02,592][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:07:03,093][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:07:03,600][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:07:04,119][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:07:04,623][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:07:05,143][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:07:05,647][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:07:06,151][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:07:06,657][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:07:07,168][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:07:07,672][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:07:08,179][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:07:08,681][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:07:09,185][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:07:09,686][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:07:10,189][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:07:10,691][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:07:11,223][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:07:11,743][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:07:12,252][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:07:12,761][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:07:13,270][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:07:13,779][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:07:14,284][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:07:14,788][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:07:15,294][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:07:15,796][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:07:16,300][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:07:16,801][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:07:17,305][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:07:17,807][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:07:18,312][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:07:18,811][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:07:19,316][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:07:19,835][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:07:20,342][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:07:20,849][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:07:21,353][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:07:21,877][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:07:22,391][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:07:22,898][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10125 tokens.
[2025-11-13 07:07:23,793][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:33
[2025-11-13 07:07:24,558][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:07:24,560][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:07:24,562][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:07:25,636][__main__][INFO] - Iteration 565 took 1m 6s (44.95% Gen, 53.44% Train). Generation: 30s, Training: 35s. Estimated remaining time: 46h 51m 30s. Estimated total time: 55h 49m 14s. Time estimates for 10 more iterations: 11m 9s, 100 more iterations: 1h 51m 38s, 500 more iterations: 9h 18m 12s.
[2025-11-13 07:07:25,639][__main__][INFO] - Starting iteration 565.
[2025-11-13 07:07:26,116][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:07:26,117][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:07:43,382][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:07:52,755][__main__][INFO] - Number of regex retries in iteration 565: 1
[2025-11-13 07:07:52,756][__main__][INFO] - agents played in iteration 565 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:07:53,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:07:53,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:07:53,616][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:07:53,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:07:53,639][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:07:53,639][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:07:54,533][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:07:54,992][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:07:55,508][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:07:57,678][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:07:58,180][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:07:58,687][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:07:59,188][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:07:59,693][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:08:00,198][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:08:00,700][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:08:01,200][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:08:01,701][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:08:02,204][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:08:02,706][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:08:03,209][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:08:03,711][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:08:04,213][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:08:04,713][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:08:05,215][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:08:05,714][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:08:06,228][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:08:06,727][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:08:07,232][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:08:07,732][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:08:08,231][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:08:08,750][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:08:09,254][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:08:09,755][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:08:10,257][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:08:10,761][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:08:11,268][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:08:11,769][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:08:12,268][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:08:12,769][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:08:13,266][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:08:13,767][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:08:14,268][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:08:14,766][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:08:15,269][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:08:15,770][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:08:16,273][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:08:16,776][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:08:17,278][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:08:17,780][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:08:18,278][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:08:18,783][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:08:19,282][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:08:19,780][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:08:20,282][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:08:20,783][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:08:21,281][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:08:21,783][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:08:22,287][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:08:22,787][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:08:23,290][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:08:23,788][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:08:24,288][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:08:24,790][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:08:25,291][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:08:25,790][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:08:26,288][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:08:26,787][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:08:27,284][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:08:27,785][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:08:28,286][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10124 tokens.
[2025-11-13 07:08:29,128][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.06%, ΔTime: 00:00:34
[2025-11-13 07:08:29,809][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:08:29,811][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:08:29,812][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:08:31,014][__main__][INFO] - Iteration 566 took 1m 4s (41.05% Gen, 57.10% Train). Generation: 26s, Training: 37s. Estimated remaining time: 45h 6m 8s. Estimated total time: 54h 4m 57s. Time estimates for 10 more iterations: 10m 48s, 100 more iterations: 1h 48m 9s, 500 more iterations: 9h 0m 49s.
[2025-11-13 07:08:31,018][__main__][INFO] - Starting iteration 566.
[2025-11-13 07:08:31,507][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:08:31,508][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:08:54,042][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:08:59,052][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:09:04,615][__main__][INFO] - Number of regex retries in iteration 566: 2
[2025-11-13 07:09:04,616][__main__][INFO] - agents played in iteration 566 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:09:05,441][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:09:05,467][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:09:05,504][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:09:05,528][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:09:05,529][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:09:05,530][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:09:06,440][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:09:06,903][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:09:07,410][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:09:07,912][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:09:08,419][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:09:08,921][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:09:09,424][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:09:09,929][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:09:10,433][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:09:10,941][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:09:11,445][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:09:11,953][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:09:12,470][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:09:12,974][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:09:13,482][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:09:13,981][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:09:14,484][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:09:14,992][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:09:15,493][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:09:15,995][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:09:16,497][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:09:16,997][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:09:17,497][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:09:17,996][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:09:18,496][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:09:19,009][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:09:19,515][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:09:20,013][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:09:20,519][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:09:21,020][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:09:21,525][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:09:22,031][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:09:22,531][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:09:23,046][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:09:23,546][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:09:24,059][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:09:24,565][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:09:25,067][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:09:25,584][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:09:26,089][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:09:26,594][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:09:27,093][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:09:27,594][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:09:28,096][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:09:28,596][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:09:29,103][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:09:29,609][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:09:30,107][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:09:30,608][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:09:31,108][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:09:31,607][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:09:32,108][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:09:32,605][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:09:33,107][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:09:33,607][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:09:34,110][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:09:34,612][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:09:35,114][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:09:35,615][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:09:36,113][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:09:36,611][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:09:37,108][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:09:37,609][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:09:38,107][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:09:38,605][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10133 tokens.
[2025-11-13 07:09:39,476][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:33
[2025-11-13 07:09:40,253][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:09:40,255][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:09:40,257][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:09:41,275][__main__][INFO] - Iteration 567 took 1m 9s (47.45% Gen, 51.08% Train). Generation: 33s, Training: 35s. Estimated remaining time: 49h 8m 26s. Estimated total time: 58h 8m 25s. Time estimates for 10 more iterations: 11m 37s, 100 more iterations: 1h 56m 16s, 500 more iterations: 9h 41m 24s.
[2025-11-13 07:09:41,278][__main__][INFO] - Starting iteration 567.
[2025-11-13 07:09:41,757][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:09:41,758][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:10:06,856][__main__][INFO] - Number of regex retries in iteration 567: 0
[2025-11-13 07:10:06,859][__main__][INFO] - agents played in iteration 567 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:10:07,707][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:10:07,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:10:07,760][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:10:07,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:10:07,786][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:10:07,786][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:10:08,692][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:10:09,151][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:10:09,666][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:10:10,166][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:10:10,688][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:10:11,193][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:10:11,698][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:10:12,202][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:10:12,703][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:10:13,207][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:10:13,708][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:10:14,209][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:10:14,711][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:10:15,212][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:10:15,715][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:10:16,219][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:10:16,722][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:10:17,227][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:10:17,730][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:10:18,231][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:10:18,732][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:10:19,233][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:10:19,737][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:10:20,238][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:10:20,740][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:10:21,243][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:10:21,742][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:10:22,240][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:10:22,753][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:10:23,253][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:10:23,775][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:10:24,274][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:10:24,773][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:10:25,276][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:10:25,776][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:10:26,286][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:10:26,785][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:10:27,289][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:10:27,789][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:10:28,286][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:10:28,787][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:10:29,287][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:10:29,785][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:10:30,298][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:10:30,800][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:10:31,301][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:10:31,806][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:10:32,307][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:10:32,809][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:10:33,311][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:10:33,815][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:10:34,317][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:10:34,820][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:10:35,321][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:10:35,826][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:10:36,330][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:10:36,838][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:10:37,340][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:10:37,840][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:10:38,343][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:10:38,846][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:10:39,349][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:10:39,856][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:10:40,356][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:10:40,871][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10050 tokens.
[2025-11-13 07:10:41,796][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.03%, ΔTime: 00:00:33
[2025-11-13 07:10:42,451][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:10:42,453][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:10:42,455][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:10:43,362][__main__][INFO] - Iteration 568 took 1m 1s (40.75% Gen, 57.78% Train). Generation: 25s, Training: 35s. Estimated remaining time: 42h 19m 16s. Estimated total time: 51h 20m 18s. Time estimates for 10 more iterations: 10m 16s, 100 more iterations: 1h 42m 40s, 500 more iterations: 8h 33m 23s.
[2025-11-13 07:10:43,365][__main__][INFO] - Starting iteration 568.
[2025-11-13 07:10:43,871][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:10:43,871][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:11:17,313][__main__][INFO] - Number of regex retries in iteration 568: 0
[2025-11-13 07:11:17,313][__main__][INFO] - agents played in iteration 568 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:11:18,150][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:11:18,173][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:11:18,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:11:18,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:11:18,234][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:11:18,235][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:11:19,131][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:11:19,605][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:11:20,116][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:11:20,624][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:11:21,130][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:11:21,637][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:11:22,142][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:11:22,653][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:11:23,159][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:11:23,676][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:11:24,176][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:11:24,683][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:11:25,186][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:11:25,690][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:11:26,201][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:11:26,705][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:11:27,208][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:11:27,712][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:11:28,213][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:11:28,718][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:11:29,223][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:11:29,724][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:11:30,238][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:11:30,742][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:11:31,245][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:11:31,746][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:11:32,250][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:11:32,761][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:11:33,262][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:11:33,763][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:11:34,272][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:11:34,775][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:11:35,288][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:11:35,796][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:11:36,299][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:11:36,811][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:11:37,315][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:11:37,817][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:11:38,318][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:11:38,819][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:11:39,322][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:11:39,825][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:11:40,326][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:11:40,832][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:11:41,332][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:11:41,836][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:11:42,336][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:11:42,840][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:11:43,351][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:11:43,850][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:11:44,351][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:11:44,853][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:11:45,352][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:11:45,852][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:11:46,353][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:11:46,852][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:11:47,358][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:11:47,860][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:11:48,359][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:11:48,871][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:11:49,372][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:11:49,896][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:11:50,397][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:11:50,897][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:11:51,404][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10124 tokens.
[2025-11-13 07:11:52,217][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.27%, ΔTime: 00:00:33
[2025-11-13 07:11:52,972][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:11:52,973][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:11:52,975][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:11:54,052][__main__][INFO] - Iteration 569 took 1m 10s (47.65% Gen, 50.81% Train). Generation: 33s, Training: 35s. Estimated remaining time: 49h 26m 53s. Estimated total time: 58h 29m 5s. Time estimates for 10 more iterations: 11m 41s, 100 more iterations: 1h 56m 58s, 500 more iterations: 9h 44m 50s.
[2025-11-13 07:11:54,054][__main__][INFO] - Starting iteration 569.
[2025-11-13 07:11:54,528][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:11:54,529][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:12:20,625][__main__][INFO] - Number of regex retries in iteration 569: 0
[2025-11-13 07:12:20,627][__main__][INFO] - agents played in iteration 569 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:12:21,685][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:12:21,714][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:12:21,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:12:21,764][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:12:21,765][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:12:21,766][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:12:22,652][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:12:23,118][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:12:23,628][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:12:24,134][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:12:24,643][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:12:25,148][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:12:25,654][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:12:26,160][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:12:26,665][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:12:27,178][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:12:27,681][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:12:28,195][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:12:28,698][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:12:29,203][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:12:29,710][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:12:30,212][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:12:30,722][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:12:31,227][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:12:31,730][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:12:32,239][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:12:32,742][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:12:33,248][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:12:33,752][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:12:34,261][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:12:34,769][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:12:35,271][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:12:35,773][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:12:36,277][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:12:36,781][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:12:37,284][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:12:37,787][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:12:38,288][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:12:38,798][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:12:39,302][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:12:39,802][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:12:40,305][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:12:40,808][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:12:41,312][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:12:41,815][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:12:42,316][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:12:42,824][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:12:43,323][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:12:43,823][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:12:44,321][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:12:44,820][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:12:45,321][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:12:45,821][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:12:46,322][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:12:46,824][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:12:47,327][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:12:47,828][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:12:48,333][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:12:48,835][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:12:49,337][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:12:49,841][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:12:50,346][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:12:50,852][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:12:51,354][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:12:51,855][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:12:52,370][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:12:52,877][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:12:53,393][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:12:53,901][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:12:54,404][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:12:54,910][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10055 tokens.
[2025-11-13 07:12:55,783][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:33
[2025-11-13 07:12:56,454][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:12:56,455][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:12:56,457][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:12:57,358][__main__][INFO] - Iteration 570 took 1m 2s (41.54% Gen, 57.03% Train). Generation: 26s, Training: 35s. Estimated remaining time: 43h 18m 17s. Estimated total time: 52h 21m 32s. Time estimates for 10 more iterations: 10m 28s, 100 more iterations: 1h 44m 43s, 500 more iterations: 8h 43m 35s.
[2025-11-13 07:12:57,361][__main__][INFO] - Starting iteration 570.
[2025-11-13 07:12:57,845][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 07:12:57,845][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:13:17,339][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:13:28,984][__main__][INFO] - Number of regex retries in iteration 570: 1
[2025-11-13 07:13:28,985][__main__][INFO] - agents played in iteration 570 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:13:29,863][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:13:29,888][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:13:29,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:13:29,937][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:13:29,937][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:13:29,938][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:13:30,813][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:13:31,285][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:13:31,793][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:13:32,302][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:13:32,822][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:13:33,327][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:13:33,832][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:13:34,336][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:13:34,837][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:13:35,342][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:13:35,843][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:13:36,351][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:13:36,852][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:13:37,353][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:13:37,854][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:13:38,355][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:13:38,857][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:13:39,358][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:13:39,861][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:13:40,362][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:13:40,859][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:13:41,356][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:13:41,858][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:13:42,354][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:13:42,852][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:13:43,355][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:13:43,852][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:13:44,351][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:13:44,853][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:13:45,353][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:13:45,856][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:13:46,354][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:13:46,854][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:13:47,355][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:13:47,855][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:13:48,354][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:13:48,855][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:13:49,355][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:13:49,858][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:13:50,356][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:13:50,857][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:13:51,356][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:13:51,853][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:13:52,354][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:13:52,858][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:13:53,360][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:13:53,860][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:13:54,356][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:13:54,853][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:13:55,355][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:13:55,854][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:13:56,352][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:13:56,848][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:13:57,345][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:13:57,845][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:13:58,344][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:13:58,841][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:13:59,341][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:13:59,841][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:14:00,342][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:14:00,839][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:14:01,338][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:14:01,842][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:14:02,345][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:14:02,845][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10129 tokens.
[2025-11-13 07:14:03,654][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 07:14:04,403][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:14:04,405][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:14:04,407][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:14:06,393][__main__][INFO] - Iteration 571 took 1m 8s (45.43% Gen, 51.67% Train). Generation: 31s, Training: 35s. Estimated remaining time: 48h 3m 2s. Estimated total time: 57h 7m 27s. Time estimates for 10 more iterations: 11m 25s, 100 more iterations: 1h 54m 14s, 500 more iterations: 9h 31m 14s.
[2025-11-13 07:14:06,395][__main__][INFO] - Starting iteration 571.
[2025-11-13 07:14:06,901][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:14:06,901][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:14:32,640][__main__][INFO] - Number of regex retries in iteration 571: 0
[2025-11-13 07:14:32,642][__main__][INFO] - agents played in iteration 571 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:14:33,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:14:33,699][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:14:33,721][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:14:33,744][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:14:33,744][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:14:33,745][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:14:34,616][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:14:35,072][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:14:35,584][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:14:36,085][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:14:36,590][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:14:37,103][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:14:37,602][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:14:38,101][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:14:38,601][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:14:39,105][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:14:39,621][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:14:40,123][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:14:40,629][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:14:41,132][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:14:41,637][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:14:42,142][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:14:42,644][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:14:43,144][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:14:43,649][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:14:44,149][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:14:44,652][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:14:45,153][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:14:45,653][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:14:46,156][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:14:46,655][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:14:47,154][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:14:47,655][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:14:48,152][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:14:48,657][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:14:49,159][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:14:49,658][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:14:50,159][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:14:50,656][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:14:51,157][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:14:51,652][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:14:52,149][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:14:52,650][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:14:53,147][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:14:53,646][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:14:54,144][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:14:54,641][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:14:55,146][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:14:55,643][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:14:56,141][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:14:56,640][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:14:57,144][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:14:57,642][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:14:58,139][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:14:58,641][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:14:59,150][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:14:59,649][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:15:00,150][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:15:00,651][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:15:01,152][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:15:01,657][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:15:02,157][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:15:02,656][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:15:03,154][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:15:03,656][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:15:04,156][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:15:04,657][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:15:05,160][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:15:05,657][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:15:06,155][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:15:06,653][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10059 tokens.
[2025-11-13 07:15:07,485][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.35%, ΔTime: 00:00:32
[2025-11-13 07:15:08,134][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:15:08,136][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:15:08,138][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:15:09,216][__main__][INFO] - Iteration 572 took 1m 2s (41.31% Gen, 56.96% Train). Generation: 25s, Training: 35s. Estimated remaining time: 42h 50m 22s. Estimated total time: 51h 55m 49s. Time estimates for 10 more iterations: 10m 23s, 100 more iterations: 1h 43m 51s, 500 more iterations: 8h 39m 18s.
[2025-11-13 07:15:09,218][__main__][INFO] - Starting iteration 572.
[2025-11-13 07:15:09,696][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:15:09,697][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:15:42,236][__main__][INFO] - Number of regex retries in iteration 572: 0
[2025-11-13 07:15:42,237][__main__][INFO] - agents played in iteration 572 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:15:43,069][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:15:43,092][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:15:43,114][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:15:43,137][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:15:43,138][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:15:43,140][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:15:44,004][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:15:44,460][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:15:44,966][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:15:45,470][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:15:45,970][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:15:46,470][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:15:46,969][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:15:47,470][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:15:47,973][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:15:48,474][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:15:48,984][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:15:49,486][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:15:49,990][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:15:50,491][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:15:50,989][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:15:51,495][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:15:51,997][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:15:52,498][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:15:52,998][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:15:53,495][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:15:53,994][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:15:54,496][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:15:54,996][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:15:55,496][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:15:56,000][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:15:56,498][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:15:56,998][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:15:57,500][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:15:58,019][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:15:58,521][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:15:59,030][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:15:59,530][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:16:00,044][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:16:00,541][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:16:01,046][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:16:01,544][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:16:02,044][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:16:02,553][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:16:03,052][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:16:03,550][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:16:04,066][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:16:04,565][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:16:05,065][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:16:05,565][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:16:06,064][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:16:06,569][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:16:07,074][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:16:07,573][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:16:08,074][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:16:08,572][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:16:09,073][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:16:09,571][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:16:10,073][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:16:10,574][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:16:11,074][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:16:11,574][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:16:12,071][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:16:12,568][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:16:13,075][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:16:13,574][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:16:14,075][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:16:14,574][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:16:15,073][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:16:15,570][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:16:16,067][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10065 tokens.
[2025-11-13 07:16:16,883][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:32
[2025-11-13 07:16:17,637][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:16:17,639][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:16:17,640][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:16:18,700][__main__][INFO] - Iteration 573 took 1m 9s (47.15% Gen, 51.31% Train). Generation: 32s, Training: 35s. Estimated remaining time: 48h 23m 36s. Estimated total time: 57h 30m 14s. Time estimates for 10 more iterations: 11m 30s, 100 more iterations: 1h 55m 0s, 500 more iterations: 9h 35m 2s.
[2025-11-13 07:16:18,702][__main__][INFO] - Starting iteration 573.
[2025-11-13 07:16:19,185][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:16:19,186][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:16:35,218][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:16:46,796][__main__][INFO] - Number of regex retries in iteration 573: 1
[2025-11-13 07:16:46,797][__main__][INFO] - agents played in iteration 573 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:16:47,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:16:47,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:16:47,666][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:16:47,689][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:16:47,689][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:16:47,690][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:16:48,566][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:16:49,027][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:16:49,532][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:16:50,034][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:16:50,543][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:16:51,049][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:16:51,563][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:16:52,064][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:16:52,563][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:16:53,076][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:16:53,577][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:16:54,083][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:16:54,582][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:16:55,085][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:16:55,593][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:16:56,095][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:16:56,593][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:16:57,096][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:16:57,599][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:16:58,102][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:16:58,605][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:16:59,105][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:16:59,611][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:17:00,109][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:17:00,608][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:17:01,107][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:17:01,605][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:17:02,107][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:17:02,606][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:17:03,104][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:17:03,603][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:17:04,101][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:17:04,647][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:17:05,148][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:17:05,652][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:17:06,153][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:17:06,652][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:17:07,153][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:17:07,656][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:17:08,156][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:17:08,671][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:17:09,169][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:17:09,670][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:17:10,173][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:17:10,676][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:17:11,179][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:17:11,677][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:17:12,173][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:17:12,671][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:17:13,168][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:17:13,667][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:17:14,167][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:17:14,663][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:17:15,170][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:17:15,673][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:17:16,173][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:17:16,672][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:17:17,168][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:17:17,665][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:17:18,165][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:17:18,660][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:17:19,175][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:17:19,670][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:17:20,166][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:17:20,669][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10098 tokens.
[2025-11-13 07:17:21,459][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 07:17:22,227][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:17:22,228][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:17:22,230][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:17:23,266][__main__][INFO] - Iteration 574 took 1m 4s (43.09% Gen, 55.29% Train). Generation: 27s, Training: 35s. Estimated remaining time: 44h 16m 23s. Estimated total time: 53h 24m 5s. Time estimates for 10 more iterations: 10m 40s, 100 more iterations: 1h 46m 48s, 500 more iterations: 8h 54m 0s.
[2025-11-13 07:17:23,268][__main__][INFO] - Starting iteration 574.
[2025-11-13 07:17:23,766][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:17:23,766][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:17:43,011][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:17:53,725][__main__][INFO] - Number of regex retries in iteration 574: 1
[2025-11-13 07:17:53,726][__main__][INFO] - agents played in iteration 574 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:17:54,587][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:17:54,611][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:17:54,634][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:17:54,657][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:17:54,658][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:17:54,658][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:17:55,509][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:17:56,061][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:17:56,571][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:17:57,073][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:17:57,571][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:17:58,079][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:17:58,582][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:17:59,088][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:17:59,589][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:18:00,092][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:18:00,592][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:18:01,096][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:18:01,596][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:18:02,099][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:18:02,606][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:18:03,108][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:18:03,609][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:18:04,111][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:18:04,612][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:18:05,112][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:18:05,615][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:18:06,116][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:18:06,622][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:18:07,123][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:18:07,631][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:18:08,132][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:18:08,633][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:18:09,144][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:18:09,644][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:18:10,144][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:18:10,641][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:18:11,140][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:18:11,659][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:18:12,156][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:18:12,653][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:18:13,154][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:18:13,652][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:18:14,162][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:18:14,660][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:18:15,158][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:18:15,670][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:18:16,167][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:18:16,666][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:18:17,164][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:18:17,660][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:18:18,169][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:18:18,668][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:18:19,168][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:18:19,667][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:18:20,167][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:18:20,670][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:18:21,169][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:18:21,667][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:18:22,167][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:18:22,668][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:18:23,167][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:18:23,669][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:18:24,167][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:18:24,698][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:18:25,199][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:18:25,700][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:18:26,197][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:18:26,694][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:18:27,194][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:18:27,689][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10121 tokens.
[2025-11-13 07:18:28,487][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:32
[2025-11-13 07:18:29,121][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:18:29,122][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:18:29,125][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:18:30,301][__main__][INFO] - Iteration 575 took 1m 6s (45.03% Gen, 53.20% Train). Generation: 29s, Training: 35s. Estimated remaining time: 46h 17m 58s. Estimated total time: 55h 26m 47s. Time estimates for 10 more iterations: 11m 5s, 100 more iterations: 1h 50m 53s, 500 more iterations: 9h 14m 27s.
[2025-11-13 07:18:30,303][__main__][INFO] - Starting iteration 575.
[2025-11-13 07:18:30,787][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:18:30,788][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:19:00,876][__main__][INFO] - Number of regex retries in iteration 575: 0
[2025-11-13 07:19:00,877][__main__][INFO] - agents played in iteration 575 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:19:01,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:19:01,761][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:19:01,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:19:01,809][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:19:01,809][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:19:01,810][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:19:02,648][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:19:03,105][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:19:03,612][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:19:04,118][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:19:04,618][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:19:05,119][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:19:05,635][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:19:06,134][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:19:06,649][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:19:07,151][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:19:07,653][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:19:08,157][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:19:08,661][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:19:09,169][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:19:09,680][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:19:10,183][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:19:10,688][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:19:11,190][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:19:11,695][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:19:12,195][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:19:12,697][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:19:13,198][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:19:13,703][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:19:14,206][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:19:14,710][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:19:15,210][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:19:15,710][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:19:16,208][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:19:16,706][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:19:17,207][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:19:17,705][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:19:18,202][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:19:18,723][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:19:19,224][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:19:19,729][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:19:20,231][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:19:20,734][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:19:21,245][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:19:21,742][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:19:22,260][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:19:22,761][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:19:23,262][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:19:23,759][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:19:24,255][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:19:24,752][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:19:25,255][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:19:25,754][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:19:26,272][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:19:26,771][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:19:27,274][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:19:27,771][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:19:28,272][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:19:28,773][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:19:29,272][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:19:29,766][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:19:30,262][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:19:30,756][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:19:31,249][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:19:31,761][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:19:32,257][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:19:32,751][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:19:33,249][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:19:33,744][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:19:34,242][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:19:34,737][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10184 tokens.
[2025-11-13 07:19:35,538][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 07:19:36,285][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:19:36,286][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:19:36,288][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:19:37,295][__main__][INFO] - Iteration 576 took 1m 6s (45.24% Gen, 53.24% Train). Generation: 30s, Training: 35s. Estimated remaining time: 46h 15m 30s. Estimated total time: 55h 25m 26s. Time estimates for 10 more iterations: 11m 5s, 100 more iterations: 1h 50m 50s, 500 more iterations: 9h 14m 14s.
[2025-11-13 07:19:37,297][__main__][INFO] - Starting iteration 576.
[2025-11-13 07:19:37,780][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:19:37,780][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:20:02,296][__main__][INFO] - Number of regex retries in iteration 576: 0
[2025-11-13 07:20:02,299][__main__][INFO] - agents played in iteration 576 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:20:03,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:20:03,227][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:20:03,250][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:20:03,273][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:20:03,273][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:20:03,275][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:20:04,111][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:20:04,567][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:20:05,072][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:20:05,573][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:20:06,073][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:20:06,573][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:20:07,079][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:20:07,578][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:20:08,078][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:20:08,586][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:20:09,091][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:20:09,594][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:20:10,106][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:20:10,605][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:20:11,108][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:20:11,609][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:20:12,110][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:20:12,611][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:20:13,112][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:20:13,617][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:20:14,120][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:20:14,622][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:20:15,126][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:20:15,629][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:20:16,130][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:20:16,630][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:20:17,131][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:20:17,633][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:20:18,133][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:20:18,632][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:20:19,138][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:20:19,636][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:20:20,136][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:20:20,636][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:20:21,136][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:20:21,636][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:20:22,138][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:20:22,641][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:20:23,140][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:20:23,641][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:20:24,153][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:20:24,656][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:20:25,153][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:20:25,659][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:20:26,155][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:20:26,656][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:20:27,157][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:20:27,654][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:20:28,154][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:20:28,651][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:20:29,148][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:20:29,646][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:20:30,144][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:20:30,645][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:20:31,143][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:20:31,645][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:20:32,153][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:20:32,651][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:20:33,155][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:20:33,656][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:20:34,157][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:20:34,658][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:20:35,157][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:20:35,658][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:20:36,160][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10005 tokens.
[2025-11-13 07:20:37,025][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:32
[2025-11-13 07:20:37,661][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:20:37,663][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:20:37,664][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:20:38,696][__main__][INFO] - Iteration 577 took 1m 0s (40.25% Gen, 58.05% Train). Generation: 24s, Training: 35s. Estimated remaining time: 41h 34m 52s. Estimated total time: 50h 45m 49s. Time estimates for 10 more iterations: 10m 9s, 100 more iterations: 1h 41m 31s, 500 more iterations: 8h 27m 38s.
[2025-11-13 07:20:38,698][__main__][INFO] - Starting iteration 577.
[2025-11-13 07:20:39,184][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:20:39,184][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:21:07,078][__main__][INFO] - Number of regex retries in iteration 577: 0
[2025-11-13 07:21:07,079][__main__][INFO] - agents played in iteration 577 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:21:07,972][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:21:08,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:21:08,028][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:21:08,051][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:21:08,052][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:21:08,053][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:21:08,816][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:21:09,270][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:21:09,805][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:21:10,305][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:21:10,806][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:21:11,312][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:21:11,812][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:21:12,315][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:21:12,817][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:21:13,318][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:21:13,823][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:21:14,324][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:21:14,827][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:21:15,331][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:21:15,835][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:21:16,338][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:21:16,839][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:21:17,343][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:21:17,853][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:21:18,354][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:21:18,856][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:21:19,362][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:21:19,867][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:21:20,375][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:21:20,874][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:21:21,376][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:21:21,878][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:21:22,380][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:21:22,890][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:21:23,397][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:21:23,900][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:21:24,404][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:21:24,908][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:21:25,406][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:21:25,905][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:21:26,405][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:21:26,918][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:21:27,417][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:21:27,912][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:21:28,408][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:21:28,907][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:21:29,405][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:21:29,901][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:21:30,399][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:21:30,916][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:21:31,413][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:21:31,911][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:21:32,421][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:21:32,919][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:21:33,416][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:21:33,917][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:21:34,412][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:21:34,910][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:21:35,406][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:21:35,902][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:21:36,413][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:21:36,910][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:21:37,411][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:21:37,911][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:21:38,411][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:21:38,915][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:21:39,415][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:21:39,916][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:21:40,426][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:21:40,929][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10073 tokens.
[2025-11-13 07:21:41,719][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 07:21:42,450][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:21:42,451][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:21:42,453][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:21:43,510][__main__][INFO] - Iteration 578 took 1m 4s (43.36% Gen, 54.99% Train). Generation: 27s, Training: 35s. Estimated remaining time: 44h 24m 18s. Estimated total time: 53h 36m 20s. Time estimates for 10 more iterations: 10m 43s, 100 more iterations: 1h 47m 12s, 500 more iterations: 8h 56m 3s.
[2025-11-13 07:21:43,512][__main__][INFO] - Starting iteration 578.
[2025-11-13 07:21:43,998][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:21:43,999][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:22:07,695][__main__][INFO] - Number of regex retries in iteration 578: 0
[2025-11-13 07:22:07,695][__main__][INFO] - agents played in iteration 578 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:22:08,480][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:22:08,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:22:08,536][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:22:08,559][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:22:08,560][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:22:08,561][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:22:09,352][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:22:09,805][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:22:10,309][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:22:10,826][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:22:11,325][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:22:11,823][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:22:12,322][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:22:12,820][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:22:13,333][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:22:13,836][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:22:14,336][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:22:14,836][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:22:15,335][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:22:15,837][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:22:16,337][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:22:16,837][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:22:17,343][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:22:17,845][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:22:18,351][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:22:18,853][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:22:19,361][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:22:19,862][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:22:20,364][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:22:20,866][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:22:21,368][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:22:21,869][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:22:22,375][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:22:22,875][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:22:23,375][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:22:23,885][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:22:24,386][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:22:24,897][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:22:25,403][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:22:25,905][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:22:26,409][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:22:26,910][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:22:27,412][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:22:27,914][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:22:28,415][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:22:28,918][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:22:29,420][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:22:29,924][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:22:30,433][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:22:30,930][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:22:31,432][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:22:31,933][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:22:32,429][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:22:32,932][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:22:33,430][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:22:33,930][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:22:34,434][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:22:34,930][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:22:35,433][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:22:35,929][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:22:36,427][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:22:36,934][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:22:37,429][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:22:37,929][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:22:38,427][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:22:38,929][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:22:39,428][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:22:39,928][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:22:40,427][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:22:40,933][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:22:41,435][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10100 tokens.
[2025-11-13 07:22:42,298][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 07:22:43,022][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:22:43,023][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:22:43,026][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:22:44,068][__main__][INFO] - Iteration 579 took 1m 0s (39.45% Gen, 58.81% Train). Generation: 23s, Training: 35s. Estimated remaining time: 40h 50m 29s. Estimated total time: 50h 3m 32s. Time estimates for 10 more iterations: 10m 0s, 100 more iterations: 1h 40m 7s, 500 more iterations: 8h 20m 35s.
[2025-11-13 07:22:44,071][__main__][INFO] - Starting iteration 579.
[2025-11-13 07:22:44,544][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:22:44,545][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:22:57,436][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:23:09,862][__main__][INFO] - Number of regex retries in iteration 579: 1
[2025-11-13 07:23:09,865][__main__][INFO] - agents played in iteration 579 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:23:10,741][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:23:10,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:23:10,822][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:23:10,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:23:10,851][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:23:10,851][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:23:11,661][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:23:12,122][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:23:12,628][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:23:13,130][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:23:13,631][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:23:14,137][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:23:14,637][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:23:15,146][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:23:15,646][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:23:16,149][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:23:16,654][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:23:17,155][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:23:17,658][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:23:18,162][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:23:18,665][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:23:19,167][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:23:19,668][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:23:20,168][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:23:20,670][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:23:21,171][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:23:21,673][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:23:22,173][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:23:22,673][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:23:23,175][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:23:23,675][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:23:24,175][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:23:24,675][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:23:25,178][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:23:25,679][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:23:26,179][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:23:26,679][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:23:27,180][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:23:27,681][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:23:28,188][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:23:28,695][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:23:29,197][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:23:29,712][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:23:30,216][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:23:30,717][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:23:31,225][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:23:31,727][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:23:32,244][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:23:32,749][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:23:33,253][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:23:33,752][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:23:34,254][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:23:34,757][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:23:35,258][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:23:35,759][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:23:36,261][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:23:36,764][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:23:37,266][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:23:37,767][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:23:38,271][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:23:38,776][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:23:39,277][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:23:39,775][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:23:40,274][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:23:40,770][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:23:41,273][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:23:41,771][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:23:42,268][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:23:42,769][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:23:43,268][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:23:43,769][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10078 tokens.
[2025-11-13 07:23:44,566][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 07:23:45,209][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:23:45,211][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:23:45,213][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:23:46,109][__main__][INFO] - Iteration 580 took 1m 1s (41.13% Gen, 57.42% Train). Generation: 25s, Training: 35s. Estimated remaining time: 42h 4m 10s. Estimated total time: 51h 18m 15s. Time estimates for 10 more iterations: 10m 15s, 100 more iterations: 1h 42m 36s, 500 more iterations: 8h 33m 2s.
[2025-11-13 07:23:46,111][__main__][INFO] - Starting iteration 580.
[2025-11-13 07:23:46,590][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 07:23:46,591][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:24:16,937][__main__][INFO] - Number of regex retries in iteration 580: 0
[2025-11-13 07:24:16,938][__main__][INFO] - agents played in iteration 580 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:24:17,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:24:17,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:24:17,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:24:17,787][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:24:17,787][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:24:17,788][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:24:18,566][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:24:19,024][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:24:19,540][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:24:20,037][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:24:20,533][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:24:21,036][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:24:21,534][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:24:22,047][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:24:22,552][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:24:23,054][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:24:23,555][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:24:24,054][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:24:24,549][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:24:25,043][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:24:25,541][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:24:26,059][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:24:26,558][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:24:27,058][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:24:27,554][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:24:28,055][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:24:28,556][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:24:29,056][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:24:29,558][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:24:30,063][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:24:30,563][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:24:31,064][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:24:31,564][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:24:32,067][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:24:32,569][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:24:33,068][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:24:33,568][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:24:34,069][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:24:34,569][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:24:35,069][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:24:35,569][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:24:36,072][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:24:36,572][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:24:37,071][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:24:37,574][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:24:38,075][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:24:38,575][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:24:39,079][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:24:39,583][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:24:40,086][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:24:40,590][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:24:41,090][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:24:41,591][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:24:42,091][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:24:42,587][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:24:43,088][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:24:43,589][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:24:44,087][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:24:44,587][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:24:45,083][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:24:45,587][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:24:46,086][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:24:46,586][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:24:47,085][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:24:47,580][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:24:48,075][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:24:48,571][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:24:49,067][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:24:49,564][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:24:50,059][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:24:50,555][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10028 tokens.
[2025-11-13 07:24:51,370][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 07:24:52,102][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:24:52,103][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:24:52,105][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:24:54,119][__main__][INFO] - Iteration 581 took 1m 7s (44.94% Gen, 52.08% Train). Generation: 30s, Training: 35s. Estimated remaining time: 47h 1m 17s. Estimated total time: 56h 16m 29s. Time estimates for 10 more iterations: 11m 15s, 100 more iterations: 1h 52m 32s, 500 more iterations: 9h 22m 44s.
[2025-11-13 07:24:54,121][__main__][INFO] - Starting iteration 581.
[2025-11-13 07:24:54,603][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:24:54,603][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:25:16,895][__main__][INFO] - Number of regex retries in iteration 581: 0
[2025-11-13 07:25:16,896][__main__][INFO] - agents played in iteration 581 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:25:17,660][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:25:17,691][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:25:17,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:25:17,742][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:25:17,742][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:25:17,743][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:25:18,541][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:25:18,996][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:25:19,501][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:25:20,000][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:25:20,502][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:25:20,997][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:25:21,495][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:25:21,996][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:25:22,495][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:25:22,990][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:25:23,493][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:25:23,996][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:25:24,491][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:25:24,987][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:25:25,483][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:25:25,979][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:25:26,477][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:25:26,975][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:25:27,473][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:25:27,976][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:25:28,481][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:25:28,988][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:25:29,490][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:25:29,994][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:25:30,496][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:25:30,996][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:25:31,497][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:25:32,000][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:25:32,499][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:25:33,001][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:25:33,501][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:25:34,000][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:25:34,498][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:25:34,998][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:25:35,501][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:25:36,000][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:25:36,501][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:25:37,024][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:25:37,526][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:25:38,031][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:25:38,536][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:25:39,032][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:25:39,544][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:25:40,038][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:25:40,535][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:25:41,039][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:25:41,538][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:25:42,033][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:25:42,537][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:25:43,035][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:25:43,546][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:25:44,044][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:25:44,541][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:25:45,041][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:25:45,537][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:25:46,033][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:25:46,534][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:25:47,030][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:25:47,543][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:25:48,040][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:25:48,537][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:25:49,054][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:25:49,554][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:25:50,053][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:25:50,551][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10111 tokens.
[2025-11-13 07:25:51,418][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 07:25:52,142][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:25:52,144][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:25:52,146][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:25:53,221][__main__][INFO] - Iteration 582 took 58s (38.03% Gen, 60.13% Train). Generation: 22s, Training: 35s. Estimated remaining time: 39h 34m 44s. Estimated total time: 48h 50m 56s. Time estimates for 10 more iterations: 9m 46s, 100 more iterations: 1h 37m 41s, 500 more iterations: 8h 8m 29s.
[2025-11-13 07:25:53,223][__main__][INFO] - Starting iteration 582.
[2025-11-13 07:25:53,690][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:25:53,690][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:26:14,463][__main__][INFO] - Number of regex retries in iteration 582: 0
[2025-11-13 07:26:14,464][__main__][INFO] - agents played in iteration 582 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:26:15,248][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:26:15,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:26:15,292][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:26:15,314][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:26:15,315][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:26:15,316][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:26:16,094][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:26:16,568][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:26:17,068][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:26:17,571][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:26:18,074][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:26:18,573][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:26:19,073][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:26:19,568][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:26:20,063][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:26:20,573][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:26:21,067][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:26:21,562][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:26:22,056][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:26:22,552][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:26:23,061][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:26:23,553][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:26:24,047][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:26:24,552][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:26:25,056][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:26:25,556][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:26:26,056][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:26:26,555][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:26:27,070][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:26:27,570][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:26:28,072][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:26:28,819][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:26:30,129][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:26:30,631][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:26:31,153][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:26:31,654][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:26:32,166][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:26:32,665][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:26:33,170][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:26:33,674][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:26:34,174][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:26:34,680][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:26:35,179][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:26:35,678][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:26:36,186][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:26:36,685][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:26:37,186][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:26:37,687][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:26:38,189][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:26:38,694][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:26:39,195][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:26:39,694][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:26:40,198][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:26:40,700][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:26:41,196][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:26:41,698][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:26:42,195][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:26:42,692][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:26:43,189][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:26:43,686][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:26:44,188][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:26:44,683][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:26:45,182][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:26:45,679][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:26:46,185][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:26:46,699][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:26:47,205][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:26:47,711][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:26:48,220][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:26:48,718][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:26:49,219][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10125 tokens.
[2025-11-13 07:26:50,082][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.06%, ΔTime: 00:00:33
[2025-11-13 07:26:50,743][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:26:50,744][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:26:50,746][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:26:51,678][__main__][INFO] - Iteration 583 took 57s (35.82% Gen, 62.57% Train). Generation: 20s, Training: 36s. Estimated remaining time: 39h 2m 16s. Estimated total time: 48h 19m 26s. Time estimates for 10 more iterations: 9m 39s, 100 more iterations: 1h 36m 38s, 500 more iterations: 8h 3m 14s.
[2025-11-13 07:26:51,683][__main__][INFO] - Starting iteration 583.
[2025-11-13 07:26:52,181][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:26:52,183][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:27:16,824][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:27:21,903][__main__][INFO] - Number of regex retries in iteration 583: 1
[2025-11-13 07:27:21,904][__main__][INFO] - agents played in iteration 583 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:27:22,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:27:22,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:27:22,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:27:22,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:27:22,888][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:27:22,889][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:27:23,680][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:27:24,132][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:27:24,642][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:27:25,142][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:27:25,641][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:27:26,153][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:27:26,649][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:27:27,151][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:27:27,652][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:27:28,150][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:27:28,652][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:27:29,153][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:27:29,650][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:27:30,153][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:27:30,652][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:27:31,155][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:27:31,654][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:27:32,155][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:27:32,666][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:27:33,166][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:27:33,668][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:27:34,166][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:27:34,666][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:27:35,185][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:27:35,684][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:27:36,185][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:27:36,690][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:27:37,193][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:27:37,696][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:27:38,198][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:27:38,698][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:27:39,199][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:27:39,701][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:27:40,204][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:27:40,702][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:27:41,204][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:27:41,708][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:27:42,208][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:27:42,713][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:27:43,212][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:27:43,711][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:27:44,215][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:27:44,714][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:27:45,214][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:27:45,713][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:27:46,221][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:27:46,732][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:27:47,232][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:27:47,730][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:27:48,230][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:27:48,735][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:27:49,256][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:27:49,775][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:27:50,272][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:27:50,770][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:27:51,269][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:27:51,771][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:27:52,275][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:27:52,777][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:27:53,275][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:27:53,788][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:27:54,290][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:27:54,790][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:27:55,288][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:27:55,789][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10011 tokens.
[2025-11-13 07:27:56,617][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 07:27:57,380][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:27:57,382][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:27:57,383][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:27:58,461][__main__][INFO] - Iteration 584 took 1m 6s (44.84% Gen, 53.53% Train). Generation: 29s, Training: 35s. Estimated remaining time: 45h 55m 46s. Estimated total time: 55h 14m 3s. Time estimates for 10 more iterations: 11m 2s, 100 more iterations: 1h 50m 28s, 500 more iterations: 9h 12m 20s.
[2025-11-13 07:27:58,463][__main__][INFO] - Starting iteration 584.
[2025-11-13 07:27:58,957][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:27:58,958][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:28:16,252][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:28:26,599][__main__][INFO] - Number of regex retries in iteration 584: 1
[2025-11-13 07:28:26,600][__main__][INFO] - agents played in iteration 584 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:28:27,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:28:27,457][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:28:27,482][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:28:27,506][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:28:27,507][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:28:27,508][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:28:28,341][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:28:28,794][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:28:29,300][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:28:29,808][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:28:30,308][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:28:30,808][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:28:31,304][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:28:31,801][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:28:32,295][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:28:32,791][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:28:33,288][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:28:33,784][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:28:34,281][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:28:34,778][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:28:35,275][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:28:35,786][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:28:36,286][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:28:36,790][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:28:37,302][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:28:37,804][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:28:38,306][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:28:38,809][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:28:39,312][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:28:39,819][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:28:40,323][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:28:40,823][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:28:41,325][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:28:41,828][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:28:42,330][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:28:42,834][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:28:43,335][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:28:43,836][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:28:44,336][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:28:44,839][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:28:45,338][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:28:45,838][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:28:46,340][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:28:46,841][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:28:47,349][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:28:47,849][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:28:48,350][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:28:48,856][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:28:49,361][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:28:49,864][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:28:50,380][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:28:50,882][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:28:51,381][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:28:51,881][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:28:52,384][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:28:52,898][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:28:53,397][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:28:53,905][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:28:54,413][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:28:54,919][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:28:55,433][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:28:55,941][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:28:56,440][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:28:56,942][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:28:57,440][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:28:57,940][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:28:58,438][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:28:58,936][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:28:59,434][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:28:59,935][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:29:00,437][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10069 tokens.
[2025-11-13 07:29:01,312][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.13%, ΔTime: 00:00:32
[2025-11-13 07:29:02,093][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:29:02,095][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:29:02,096][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:29:03,178][__main__][INFO] - Iteration 585 took 1m 4s (43.04% Gen, 55.27% Train). Generation: 27s, Training: 35s. Estimated remaining time: 44h 11m 45s. Estimated total time: 53h 31m 7s. Time estimates for 10 more iterations: 10m 42s, 100 more iterations: 1h 47m 2s, 500 more iterations: 8h 55m 11s.
[2025-11-13 07:29:03,181][__main__][INFO] - Starting iteration 585.
[2025-11-13 07:29:03,670][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:29:03,671][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:29:26,345][__main__][INFO] - Number of regex retries in iteration 585: 0
[2025-11-13 07:29:26,347][__main__][INFO] - agents played in iteration 585 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:29:27,197][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:29:27,225][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:29:27,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:29:27,275][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:29:27,275][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:29:27,276][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:29:28,112][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:29:28,573][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:29:29,077][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:29:29,575][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:29:30,078][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:29:30,577][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:29:31,083][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:29:31,581][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:29:32,086][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:29:32,599][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:29:33,094][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:29:33,596][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:29:34,091][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:29:34,589][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:29:35,088][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:29:35,585][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:29:36,082][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:29:36,576][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:29:37,072][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:29:37,572][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:29:38,068][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:29:38,563][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:29:39,065][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:29:39,564][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:29:40,060][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:29:40,555][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:29:41,058][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:29:41,561][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:29:42,060][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:29:42,558][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:29:43,056][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:29:43,553][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:29:44,053][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:29:44,555][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:29:45,056][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:29:45,557][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:29:46,060][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:29:46,564][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:29:47,066][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:29:47,570][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:29:48,074][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:29:48,575][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:29:49,075][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:29:49,591][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:29:50,095][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:29:50,595][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:29:51,095][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:29:51,595][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:29:52,113][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:29:52,616][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:29:53,119][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:29:53,620][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:29:54,120][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:29:54,628][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:29:55,130][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:29:55,632][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:29:56,138][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:29:56,636][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:29:57,137][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:29:57,637][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:29:58,139][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:29:58,638][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:29:59,136][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:29:59,636][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:30:00,139][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10134 tokens.
[2025-11-13 07:30:01,036][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:32
[2025-11-13 07:30:01,715][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:30:01,717][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:30:01,719][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:30:02,651][__main__][INFO] - Iteration 586 took 58s (38.45% Gen, 59.97% Train). Generation: 22s, Training: 35s. Estimated remaining time: 39h 48m 43s. Estimated total time: 49h 9m 4s. Time estimates for 10 more iterations: 9m 49s, 100 more iterations: 1h 38m 18s, 500 more iterations: 8h 11m 30s.
[2025-11-13 07:30:02,654][__main__][INFO] - Starting iteration 586.
[2025-11-13 07:30:03,136][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:30:03,137][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:30:19,832][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:30:29,752][__main__][INFO] - Number of regex retries in iteration 586: 1
[2025-11-13 07:30:29,752][__main__][INFO] - agents played in iteration 586 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:30:30,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:30:30,616][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:30:30,639][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:30:30,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:30:30,662][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:30:30,663][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:30:31,521][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:30:31,981][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:30:32,484][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:30:32,987][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:30:33,483][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:30:33,984][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:30:34,479][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:30:34,975][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:30:35,473][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:30:35,968][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:30:36,464][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:30:36,963][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:30:37,457][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:30:37,957][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:30:38,451][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:30:38,947][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:30:39,444][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:30:39,940][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:30:40,436][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:30:40,938][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:30:41,434][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:30:41,936][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:30:42,435][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:30:42,936][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:30:43,430][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:30:43,928][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:30:44,430][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:30:44,929][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:30:45,424][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:30:45,922][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:30:46,421][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:30:46,918][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:30:47,413][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:30:47,911][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:30:48,421][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:30:48,918][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:30:49,414][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:30:49,916][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:30:50,412][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:30:50,912][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:30:51,409][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:30:51,906][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:30:52,416][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:30:52,912][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:30:53,408][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:30:53,906][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:30:54,404][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:30:54,906][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:30:55,402][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:30:55,901][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:30:56,418][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:30:56,923][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:30:57,423][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:30:57,923][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:30:58,423][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:30:58,932][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:30:59,432][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:30:59,932][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:31:00,438][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:31:00,936][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:31:01,437][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:31:01,937][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:31:02,437][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:31:02,946][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:31:03,449][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10118 tokens.
[2025-11-13 07:31:04,375][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 07:31:05,117][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:31:05,118][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:31:05,120][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:31:06,252][__main__][INFO] - Iteration 587 took 1m 3s (42.17% Gen, 56.04% Train). Generation: 26s, Training: 35s. Estimated remaining time: 43h 14m 26s. Estimated total time: 52h 35m 50s. Time estimates for 10 more iterations: 10m 31s, 100 more iterations: 1h 45m 11s, 500 more iterations: 8h 45m 58s.
[2025-11-13 07:31:06,255][__main__][INFO] - Starting iteration 587.
[2025-11-13 07:31:06,777][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:31:06,777][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:31:29,819][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:31:31,186][__main__][INFO] - Number of regex retries in iteration 587: 1
[2025-11-13 07:31:31,186][__main__][INFO] - agents played in iteration 587 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:31:31,998][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:31:32,026][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:31:32,052][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:31:32,076][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:31:32,077][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:31:32,078][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:31:32,885][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:31:33,341][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:31:33,844][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:31:34,343][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:31:34,846][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:31:35,343][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:31:35,844][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:31:36,341][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:31:36,838][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:31:37,336][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:31:37,832][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:31:38,341][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:31:38,835][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:31:39,331][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:31:39,832][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:31:40,330][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:31:40,833][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:31:41,328][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:31:41,822][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:31:42,318][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:31:42,816][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:31:43,311][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:31:43,813][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:31:44,315][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:31:44,831][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:31:45,326][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:31:45,821][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:31:46,317][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:31:46,813][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:31:47,315][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:31:47,814][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:31:48,309][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:31:48,808][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:31:49,307][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:31:49,803][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:31:50,308][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:31:50,804][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:31:51,306][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:31:51,806][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:31:52,305][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:31:52,807][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:31:53,305][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:31:53,805][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:31:54,304][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:31:54,802][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:31:55,308][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:31:55,807][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:31:56,306][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:31:56,806][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:31:57,306][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:31:57,809][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:31:58,307][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:31:58,808][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:31:59,322][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:31:59,828][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:32:00,331][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:32:00,836][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:32:01,339][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:32:01,850][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:32:02,353][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:32:02,855][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:32:03,361][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:32:03,863][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:32:04,366][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:32:04,869][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10179 tokens.
[2025-11-13 07:32:05,799][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 07:32:06,537][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:32:06,539][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:32:06,540][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:32:07,542][__main__][INFO] - Iteration 588 took 1m 0s (40.17% Gen, 58.18% Train). Generation: 24s, Training: 35s. Estimated remaining time: 41h 15m 51s. Estimated total time: 50h 38m 17s. Time estimates for 10 more iterations: 10m 7s, 100 more iterations: 1h 41m 16s, 500 more iterations: 8h 26m 22s.
[2025-11-13 07:32:07,544][__main__][INFO] - Starting iteration 588.
[2025-11-13 07:32:08,022][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:32:08,022][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:32:36,106][__main__][INFO] - Number of regex retries in iteration 588: 0
[2025-11-13 07:32:36,108][__main__][INFO] - agents played in iteration 588 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:32:36,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:32:36,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:32:36,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:32:37,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:32:37,011][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:32:37,012][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:32:37,808][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:32:38,265][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:32:38,770][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:32:39,269][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:32:39,774][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:32:40,274][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:32:40,781][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:32:41,285][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:32:41,792][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:32:42,291][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:32:42,788][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:32:43,283][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:32:43,784][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:32:44,281][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:32:44,776][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:32:45,270][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:32:45,767][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:32:46,262][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:32:46,758][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:32:47,257][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:32:47,751][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:32:48,259][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:32:48,754][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:32:49,250][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:32:49,748][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:32:50,245][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:32:50,741][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:32:51,241][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:32:51,736][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:32:52,232][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:32:52,728][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:32:53,224][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:32:53,735][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:32:54,229][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:32:54,723][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:32:55,229][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:32:55,725][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:32:56,225][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:32:56,718][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:32:57,212][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:32:57,708][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:32:58,203][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:32:58,697][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:32:59,192][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:32:59,687][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:33:00,184][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:33:00,679][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:33:01,173][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:33:01,673][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:33:02,170][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:33:02,672][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:33:03,175][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:33:03,679][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:33:04,185][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:33:04,689][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:33:05,198][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:33:05,702][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:33:06,201][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:33:06,702][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:33:07,202][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:33:07,704][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:33:08,203][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:33:08,704][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:33:09,205][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:33:09,703][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9996 tokens.
[2025-11-13 07:33:10,633][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 07:33:11,280][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:33:11,282][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:33:11,285][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:33:12,260][__main__][INFO] - Iteration 589 took 1m 4s (43.72% Gen, 54.76% Train). Generation: 28s, Training: 35s. Estimated remaining time: 44h 8m 25s. Estimated total time: 53h 31m 56s. Time estimates for 10 more iterations: 10m 42s, 100 more iterations: 1h 47m 3s, 500 more iterations: 8h 55m 19s.
[2025-11-13 07:33:12,263][__main__][INFO] - Starting iteration 589.
[2025-11-13 07:33:12,766][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:33:12,767][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:33:29,376][mllm.models.large_language_model_local][WARNING] - Response Proposal: x hats, y books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:33:33,769][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:33:41,278][__main__][INFO] - Number of regex retries in iteration 589: 2
[2025-11-13 07:33:41,279][__main__][INFO] - agents played in iteration 589 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:33:42,125][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:33:42,150][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:33:42,176][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:33:42,198][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:33:42,199][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:33:42,200][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:33:42,981][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:33:43,432][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:33:43,950][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:33:44,451][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:33:44,968][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:33:45,469][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:33:45,969][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:33:46,472][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:33:46,967][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:33:47,472][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:33:47,968][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:33:48,466][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:33:48,964][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:33:49,462][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:33:49,958][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:33:50,456][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:33:50,952][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:33:51,463][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:33:51,957][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:33:52,450][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:33:52,964][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:33:53,457][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:33:53,951][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:33:54,449][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:33:54,944][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:33:55,466][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:33:55,964][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:33:56,459][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:33:56,955][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:33:57,451][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:33:57,954][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:33:58,448][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:33:58,944][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:33:59,441][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:33:59,938][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:34:00,432][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:34:00,928][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:34:01,423][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:34:01,918][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:34:02,411][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:34:02,908][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:34:03,406][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:34:03,901][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:34:04,395][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:34:04,890][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:34:05,384][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:34:05,883][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:34:06,377][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:34:06,872][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:34:07,382][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:34:07,881][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:34:08,383][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:34:08,885][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:34:09,385][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:34:09,890][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:34:10,391][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:34:10,891][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:34:11,395][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:34:11,894][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:34:12,393][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:34:12,892][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:34:13,393][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:34:13,901][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:34:14,399][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:34:14,899][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10065 tokens.
[2025-11-13 07:34:15,782][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:32
[2025-11-13 07:34:16,548][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:34:16,550][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:34:16,551][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:34:17,578][__main__][INFO] - Iteration 590 took 1m 4s (43.99% Gen, 54.42% Train). Generation: 28s, Training: 35s. Estimated remaining time: 44h 36m 1s. Estimated total time: 54h 0m 37s. Time estimates for 10 more iterations: 10m 48s, 100 more iterations: 1h 48m 1s, 500 more iterations: 9h 0m 6s.
[2025-11-13 07:34:17,582][__main__][INFO] - Starting iteration 590.
[2025-11-13 07:34:18,082][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 07:34:18,082][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:34:38,533][__main__][INFO] - Number of regex retries in iteration 590: 0
[2025-11-13 07:34:38,533][__main__][INFO] - agents played in iteration 590 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:34:39,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:34:39,329][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:34:39,351][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:34:39,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:34:39,374][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:34:39,375][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:34:40,252][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:34:40,716][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:34:41,228][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:34:41,727][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:34:42,227][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:34:42,723][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:34:43,221][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:34:43,723][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:34:44,224][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:34:44,721][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:34:45,215][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:34:45,711][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:34:46,209][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:34:46,706][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:34:47,203][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:34:47,699][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:34:48,193][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:34:48,689][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:34:49,187][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:34:49,697][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:34:50,192][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:34:50,689][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:34:51,185][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:34:51,684][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:34:52,185][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:34:52,682][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:34:53,178][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:34:53,694][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:34:54,191][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:34:54,690][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:34:55,187][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:34:55,685][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:34:56,192][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:34:56,687][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:34:57,183][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:34:57,681][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:34:58,176][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:34:58,676][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:34:59,175][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:34:59,671][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:35:00,181][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:35:00,678][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:35:01,175][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:35:01,675][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:35:02,172][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:35:02,669][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:35:03,166][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:35:03,666][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:35:04,185][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:35:04,682][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:35:05,179][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:35:05,679][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:35:06,178][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:35:06,684][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:35:07,181][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:35:07,677][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:35:08,179][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:35:08,679][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:35:09,176][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:35:09,677][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:35:10,179][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:35:10,700][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:35:11,204][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:35:11,704][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:35:12,203][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10146 tokens.
[2025-11-13 07:35:13,118][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 07:35:13,853][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:35:13,855][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:35:13,857][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:35:16,218][__main__][INFO] - Iteration 591 took 58s (35.18% Gen, 60.76% Train). Generation: 20s, Training: 35s. Estimated remaining time: 39h 1m 15s. Estimated total time: 48h 26m 50s. Time estimates for 10 more iterations: 9m 41s, 100 more iterations: 1h 36m 53s, 500 more iterations: 8h 4m 28s.
[2025-11-13 07:35:16,428][__main__][INFO] - Starting iteration 591.
[2025-11-13 07:35:17,610][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:35:17,610][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:35:49,648][__main__][INFO] - Number of regex retries in iteration 591: 0
[2025-11-13 07:35:49,649][__main__][INFO] - agents played in iteration 591 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:35:50,464][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:35:50,493][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:35:50,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:35:50,544][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:35:50,544][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:35:50,545][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:35:51,336][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:35:51,797][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:35:52,297][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:35:52,796][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:35:53,292][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:35:53,794][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:35:54,293][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:35:54,790][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:35:55,289][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:35:55,786][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:35:56,283][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:35:56,787][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:35:57,288][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:35:57,801][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:35:58,297][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:35:58,792][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:35:59,292][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:35:59,788][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:36:00,292][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:36:00,791][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:36:01,287][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:36:01,804][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:36:02,299][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:36:02,796][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:36:03,292][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:36:03,791][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:36:04,310][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:36:04,806][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:36:05,302][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:36:05,804][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:36:06,301][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:36:06,806][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:36:07,303][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:36:07,799][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:36:08,301][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:36:08,800][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:36:09,295][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:36:09,792][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:36:10,289][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:36:10,809][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:36:11,306][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:36:11,804][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:36:12,302][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:36:12,796][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:36:13,304][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:36:13,799][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:36:14,293][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:36:14,791][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:36:15,287][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:36:15,782][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:36:16,277][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:36:16,776][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:36:17,288][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:36:17,789][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:36:18,291][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:36:18,795][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:36:19,297][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:36:19,809][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:36:20,309][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:36:20,811][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:36:21,315][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:36:21,817][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:36:22,322][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:36:22,821][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:36:23,320][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10065 tokens.
[2025-11-13 07:36:24,270][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:32
[2025-11-13 07:36:24,912][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:36:24,914][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:36:24,917][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:36:26,082][__main__][INFO] - Iteration 592 took 1m 8s (46.79% Gen, 51.51% Train). Generation: 32s, Training: 35s. Estimated remaining time: 47h 36m 55s. Estimated total time: 57h 3m 40s. Time estimates for 10 more iterations: 11m 24s, 100 more iterations: 1h 54m 7s, 500 more iterations: 9h 30m 36s.
[2025-11-13 07:36:26,084][__main__][INFO] - Starting iteration 592.
[2025-11-13 07:36:26,558][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:36:26,559][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:36:55,176][__main__][INFO] - Number of regex retries in iteration 592: 0
[2025-11-13 07:36:55,177][__main__][INFO] - agents played in iteration 592 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:36:56,149][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:36:56,175][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:36:56,199][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:36:56,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:36:56,223][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:36:56,224][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:36:56,992][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:36:57,448][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:36:57,952][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:36:58,452][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:36:58,952][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:36:59,455][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:36:59,953][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:37:00,453][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:37:00,949][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:37:01,444][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:37:01,943][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:37:02,438][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:37:02,937][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:37:03,433][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:37:03,931][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:37:04,433][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:37:04,929][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:37:05,426][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:37:05,929][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:37:06,426][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:37:06,923][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:37:07,418][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:37:07,913][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:37:08,423][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:37:08,922][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:37:09,421][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:37:09,915][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:37:10,411][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:37:10,907][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:37:11,411][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:37:11,908][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:37:12,422][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:37:12,918][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:37:13,416][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:37:13,916][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:37:14,412][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:37:14,920][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:37:15,416][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:37:15,912][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:37:16,410][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:37:16,907][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:37:17,407][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:37:17,912][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:37:18,417][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:37:18,936][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:37:19,440][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:37:19,941][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:37:20,442][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:37:20,943][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:37:21,451][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:37:21,953][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:37:22,456][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:37:22,961][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:37:23,461][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:37:23,962][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:37:24,464][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:37:24,967][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:37:25,471][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:37:25,973][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:37:26,474][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:37:26,980][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:37:27,483][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:37:27,983][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:37:28,484][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:37:28,985][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10080 tokens.
[2025-11-13 07:37:29,913][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 07:37:30,665][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:37:30,667][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:37:30,668][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:37:31,793][__main__][INFO] - Iteration 593 took 1m 5s (43.87% Gen, 54.41% Train). Generation: 28s, Training: 35s. Estimated remaining time: 44h 53m 57s. Estimated total time: 54h 21m 47s. Time estimates for 10 more iterations: 10m 52s, 100 more iterations: 1h 48m 43s, 500 more iterations: 9h 3m 37s.
[2025-11-13 07:37:31,796][__main__][INFO] - Starting iteration 593.
[2025-11-13 07:37:32,271][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:37:32,272][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:38:10,227][__main__][INFO] - Number of regex retries in iteration 593: 0
[2025-11-13 07:38:10,229][__main__][INFO] - agents played in iteration 593 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:38:11,072][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:38:11,102][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:38:11,129][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:38:11,152][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:38:11,153][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:38:11,154][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:38:11,934][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:38:12,387][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:38:12,885][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:38:13,389][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:38:13,883][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:38:14,380][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:38:14,880][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:38:15,375][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:38:15,876][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:38:16,374][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:38:16,876][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:38:17,376][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:38:17,876][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:38:18,381][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:38:18,884][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:38:19,381][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:38:19,884][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:38:20,385][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:38:20,888][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:38:21,389][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:38:21,893][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:38:22,397][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:38:22,899][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:38:23,403][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:38:23,910][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:38:24,410][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:38:24,918][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:38:25,418][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:38:25,919][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:38:26,421][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:38:26,921][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:38:27,423][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:38:27,923][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:38:28,426][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:38:28,941][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:38:29,442][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:38:29,944][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:38:30,445][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:38:30,948][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:38:31,453][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:38:31,955][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:38:32,456][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:38:32,959][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:38:33,461][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:38:33,963][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:38:34,463][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:38:34,965][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:38:35,470][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:38:35,972][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:38:36,473][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:38:36,976][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:38:37,481][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:38:37,982][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:38:38,483][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:38:38,993][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:38:39,494][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:38:39,995][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:38:40,498][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:38:41,031][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:38:41,542][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:38:42,051][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:38:42,554][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:38:43,054][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:38:43,555][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:38:44,059][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10078 tokens.
[2025-11-13 07:38:44,880][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.36%, ΔTime: 00:00:32
[2025-11-13 07:38:45,515][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:38:45,516][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:38:45,518][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:38:46,439][__main__][INFO] - Iteration 594 took 1m 14s (51.18% Gen, 47.58% Train). Generation: 37s, Training: 35s. Estimated remaining time: 52h 19m 23s. Estimated total time: 61h 48m 28s. Time estimates for 10 more iterations: 12m 21s, 100 more iterations: 2h 3m 36s, 500 more iterations: 10h 18m 4s.
[2025-11-13 07:38:46,442][__main__][INFO] - Starting iteration 594.
[2025-11-13 07:38:46,918][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:38:46,919][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:39:13,155][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:39:15,758][__main__][INFO] - Number of regex retries in iteration 594: 1
[2025-11-13 07:39:15,759][__main__][INFO] - agents played in iteration 594 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:39:16,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:39:16,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:39:16,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:39:16,695][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:39:16,695][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:39:16,696][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:39:17,467][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:39:17,943][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:39:18,443][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:39:18,938][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:39:19,439][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:39:19,935][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:39:20,450][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:39:20,947][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:39:21,444][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:39:21,943][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:39:22,444][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:39:22,951][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:39:23,451][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:39:23,953][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:39:24,463][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:39:24,960][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:39:25,460][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:39:25,965][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:39:26,467][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:39:26,976][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:39:27,476][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:39:27,976][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:39:28,480][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:39:28,980][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:39:29,482][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:39:29,988][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:39:30,488][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:39:30,996][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:39:31,498][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:39:32,001][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:39:32,503][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:39:33,007][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:39:33,506][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:39:34,006][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:39:34,507][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:39:35,007][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:39:35,512][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:39:36,015][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:39:36,522][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:39:37,024][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:39:37,534][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:39:38,034][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:39:38,535][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:39:39,041][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:39:39,545][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:39:40,049][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:39:40,555][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:39:41,053][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:39:41,569][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:39:42,074][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:39:42,575][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:39:43,079][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:39:43,584][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:39:44,090][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:39:44,591][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:39:45,093][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:39:45,601][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:39:46,102][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:39:46,635][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:39:47,148][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:39:47,652][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:39:48,168][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:39:48,668][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:39:49,169][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:39:49,673][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10103 tokens.
[2025-11-13 07:39:50,505][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 07:39:51,245][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:39:51,247][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:39:51,249][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:39:52,346][__main__][INFO] - Iteration 595 took 1m 5s (44.08% Gen, 54.24% Train). Generation: 28s, Training: 35s. Estimated remaining time: 45h 1m 16s. Estimated total time: 54h 31m 26s. Time estimates for 10 more iterations: 10m 54s, 100 more iterations: 1h 49m 2s, 500 more iterations: 9h 5m 14s.
[2025-11-13 07:39:52,349][__main__][INFO] - Starting iteration 595.
[2025-11-13 07:39:52,826][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:39:52,826][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:40:01,485][mllm.models.large_language_model_local][WARNING] - Response Proposal: x hats, y books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:40:11,799][__main__][INFO] - Number of regex retries in iteration 595: 1
[2025-11-13 07:40:11,800][__main__][INFO] - agents played in iteration 595 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:40:12,579][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:40:12,610][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:40:12,637][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:40:12,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:40:12,662][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:40:12,663][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:40:13,423][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:40:13,877][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:40:14,379][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:40:14,876][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:40:15,379][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:40:15,876][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:40:16,375][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:40:16,870][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:40:17,368][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:40:17,869][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:40:18,364][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:40:18,859][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:40:19,358][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:40:19,859][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:40:20,354][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:40:20,851][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:40:21,351][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:40:21,866][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:40:22,368][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:40:22,866][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:40:23,366][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:40:23,866][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:40:24,379][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:40:24,877][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:40:25,375][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:40:25,872][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:40:26,370][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:40:26,873][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:40:27,369][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:40:27,865][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:40:28,362][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:40:28,860][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:40:29,356][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:40:29,850][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:40:30,345][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:40:30,858][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:40:31,355][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:40:31,850][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:40:32,356][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:40:32,855][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:40:33,364][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:40:33,869][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:40:34,372][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:40:34,875][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:40:35,379][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:40:35,882][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:40:36,388][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:40:36,892][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:40:37,398][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:40:37,898][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:40:38,399][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:40:38,899][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:40:39,403][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:40:39,905][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:40:40,409][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:40:40,911][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:40:41,411][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:40:42,433][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:40:43,743][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:40:44,263][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:40:44,764][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:40:45,267][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:40:45,772][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:40:46,275][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:40:46,783][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10194 tokens.
[2025-11-13 07:40:47,637][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:34
[2025-11-13 07:40:48,322][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:40:48,324][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:40:48,326][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:40:49,524][__main__][INFO] - Iteration 596 took 56s (33.46% Gen, 64.42% Train). Generation: 18s, Training: 36s. Estimated remaining time: 37h 43m 49s. Estimated total time: 47h 14m 57s. Time estimates for 10 more iterations: 9m 26s, 100 more iterations: 1h 34m 29s, 500 more iterations: 7h 52m 29s.
[2025-11-13 07:40:49,527][__main__][INFO] - Starting iteration 596.
[2025-11-13 07:40:50,009][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:40:50,010][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:41:22,650][__main__][INFO] - Number of regex retries in iteration 596: 0
[2025-11-13 07:41:22,651][__main__][INFO] - agents played in iteration 596 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:41:23,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:41:23,499][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:41:23,536][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:41:23,559][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:41:23,560][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:41:23,560][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:41:24,335][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:41:24,789][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:41:25,290][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:41:25,786][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:41:26,293][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:41:26,789][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:41:27,286][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:41:27,790][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:41:28,292][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:41:28,790][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:41:29,288][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:41:29,784][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:41:30,302][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:41:30,801][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:41:31,301][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:41:31,801][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:41:32,301][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:41:32,801][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:41:33,307][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:41:33,810][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:41:34,321][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:41:34,820][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:41:35,326][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:41:35,828][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:41:36,332][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:41:36,836][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:41:37,337][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:41:37,848][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:41:38,355][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:41:38,857][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:41:39,359][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:41:39,864][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:41:40,367][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:41:40,874][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:41:41,375][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:41:41,881][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:41:42,401][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:41:42,910][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:41:43,422][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:41:43,922][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:41:44,432][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:41:44,935][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:41:45,436][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:41:45,937][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:41:46,438][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:41:46,939][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:41:47,442][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:41:47,941][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:41:48,449][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:41:48,950][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:41:49,454][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:41:49,959][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:41:50,462][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:41:50,964][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:41:51,477][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:41:51,982][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:41:52,498][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:41:52,999][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:41:53,499][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:41:54,002][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:41:54,506][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:41:55,009][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:41:55,514][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:41:56,051][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:41:56,553][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10098 tokens.
[2025-11-13 07:41:57,381][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:33
[2025-11-13 07:41:58,171][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:41:58,173][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:41:58,176][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:41:59,266][__main__][INFO] - Iteration 597 took 1m 9s (47.13% Gen, 51.29% Train). Generation: 32s, Training: 35s. Estimated remaining time: 48h 10m 34s. Estimated total time: 57h 42m 51s. Time estimates for 10 more iterations: 11m 32s, 100 more iterations: 1h 55m 25s, 500 more iterations: 9h 37m 8s.
[2025-11-13 07:41:59,268][__main__][INFO] - Starting iteration 597.
[2025-11-13 07:41:59,763][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:41:59,765][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:42:27,150][__main__][INFO] - Number of regex retries in iteration 597: 0
[2025-11-13 07:42:27,150][__main__][INFO] - agents played in iteration 597 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:42:28,020][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:42:28,047][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:42:28,073][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:42:28,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:42:28,097][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:42:28,098][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:42:28,878][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:42:29,331][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:42:29,835][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:42:30,334][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:42:30,838][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:42:31,334][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:42:31,831][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:42:32,329][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:42:32,830][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:42:33,332][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:42:33,833][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:42:34,329][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:42:34,830][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:42:35,327][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:42:35,827][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:42:36,322][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:42:36,819][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:42:37,319][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:42:37,817][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:42:38,316][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:42:38,818][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:42:39,317][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:42:39,818][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:42:40,314][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:42:40,817][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:42:41,325][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:42:41,830][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:42:42,338][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:42:42,839][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:42:43,344][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:42:43,851][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:42:44,356][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:42:44,862][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:42:45,368][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:42:45,874][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:42:46,378][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:42:46,880][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:42:47,385][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:42:47,908][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:42:48,413][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:42:48,919][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:42:49,417][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:42:49,918][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:42:50,430][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:42:50,931][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:42:51,436][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:42:51,939][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:42:52,440][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:42:52,944][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:42:53,450][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:42:53,952][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:42:54,457][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:42:54,957][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:42:55,459][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:42:55,963][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:42:56,469][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:42:56,971][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:42:57,486][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:42:57,984][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:42:58,489][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:42:58,990][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:42:59,491][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:43:00,007][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:43:00,507][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:43:01,009][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10107 tokens.
[2025-11-13 07:43:01,849][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 07:43:02,649][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:43:02,651][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:43:02,652][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:43:03,684][__main__][INFO] - Iteration 598 took 1m 3s (42.84% Gen, 55.54% Train). Generation: 27s, Training: 35s. Estimated remaining time: 43h 42m 43s. Estimated total time: 53h 16m 5s. Time estimates for 10 more iterations: 10m 39s, 100 more iterations: 1h 46m 32s, 500 more iterations: 8h 52m 40s.
[2025-11-13 07:43:03,686][__main__][INFO] - Starting iteration 598.
[2025-11-13 07:43:04,167][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:43:04,167][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:43:28,263][__main__][INFO] - Number of regex retries in iteration 598: 0
[2025-11-13 07:43:28,264][__main__][INFO] - agents played in iteration 598 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:43:29,197][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:43:29,224][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:43:29,249][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:43:29,273][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:43:29,274][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:43:29,274][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:43:30,049][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:43:30,500][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:43:31,005][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:43:31,514][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:43:32,011][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:43:32,508][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:43:33,007][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:43:33,503][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:43:34,002][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:43:34,498][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:43:34,995][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:43:35,503][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:43:36,000][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:43:36,496][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:43:37,999][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:43:37,503][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:43:38,006][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:43:38,502][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:43:38,997][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:43:39,514][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:43:40,009][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:43:40,503][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:43:41,005][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:43:41,498][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:43:41,998][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:43:42,496][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:43:42,991][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:43:43,497][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:43:44,000][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:43:44,501][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:43:45,007][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:43:45,507][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:43:46,010][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:43:46,514][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:43:47,019][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:43:47,538][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:43:48,040][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:43:48,546][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:43:49,052][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:43:49,556][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:43:50,145][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:43:51,571][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:43:52,109][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:43:52,640][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:43:53,139][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:43:53,642][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:43:54,153][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:43:54,655][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:43:55,156][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:43:55,659][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:43:56,161][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:43:56,668][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:43:57,171][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:43:57,676][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:43:58,178][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:43:58,683][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:43:59,185][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:43:59,690][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:44:00,195][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:44:00,699][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:44:01,202][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:44:01,704][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:44:02,206][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:44:02,713][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:44:03,234][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10146 tokens.
[2025-11-13 07:44:04,175][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:34
[2025-11-13 07:44:04,881][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:44:04,882][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:44:04,884][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:44:05,999][__main__][INFO] - Iteration 599 took 1m 1s (38.97% Gen, 59.22% Train). Generation: 24s, Training: 36s. Estimated remaining time: 41h 57m 14s. Estimated total time: 51h 31m 38s. Time estimates for 10 more iterations: 10m 18s, 100 more iterations: 1h 43m 3s, 500 more iterations: 8h 35m 16s.
[2025-11-13 07:44:06,001][__main__][INFO] - Starting iteration 599.
[2025-11-13 07:44:06,487][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:44:06,487][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:44:38,374][__main__][INFO] - Number of regex retries in iteration 599: 0
[2025-11-13 07:44:38,375][__main__][INFO] - agents played in iteration 599 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:44:39,301][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:44:39,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:44:39,350][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:44:39,372][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:44:39,373][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:44:39,374][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:44:40,152][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:44:40,603][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:44:41,106][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:44:41,617][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:44:42,114][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:44:42,611][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:44:43,118][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:44:43,613][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:44:44,124][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:44:44,619][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:44:45,116][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:44:45,629][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:44:46,122][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:44:46,618][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:44:47,114][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:44:47,610][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:44:48,123][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:44:48,617][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:44:49,111][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:44:49,612][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:44:50,109][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:44:50,609][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:44:51,104][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:44:51,604][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:44:52,113][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:44:52,614][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:44:53,115][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:44:53,622][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:44:54,124][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:44:54,640][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:44:55,138][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:44:55,648][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:44:56,150][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:44:56,655][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:44:57,158][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:44:57,657][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:44:58,158][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:44:58,658][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:44:59,160][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:44:59,663][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:45:00,167][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:45:00,670][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:45:01,171][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:45:01,671][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:45:02,172][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:45:02,676][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:45:03,176][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:45:03,679][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:45:04,182][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:45:04,683][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:45:05,186][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:45:05,687][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:45:06,190][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:45:06,699][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:45:07,201][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:45:07,708][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:45:08,215][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:45:08,720][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:45:09,226][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:45:09,727][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:45:10,231][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:45:10,735][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:45:11,238][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:45:11,749][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:45:12,251][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10101 tokens.
[2025-11-13 07:45:13,137][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 07:45:13,892][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:45:13,893][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:45:13,895][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:45:14,927][__main__][INFO] - Iteration 600 took 1m 8s (46.59% Gen, 51.90% Train). Generation: 31s, Training: 35s. Estimated remaining time: 47h 26m 29s. Estimated total time: 57h 2m 2s. Time estimates for 10 more iterations: 11m 24s, 100 more iterations: 1h 54m 4s, 500 more iterations: 9h 30m 20s.
[2025-11-13 07:45:14,929][__main__][INFO] - Starting iteration 600.
[2025-11-13 07:45:15,422][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 07:45:15,422][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:45:38,489][__main__][INFO] - Number of regex retries in iteration 600: 0
[2025-11-13 07:45:38,489][__main__][INFO] - agents played in iteration 600 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:45:39,414][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:45:39,441][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:45:39,467][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:45:39,490][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:45:39,490][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:45:39,492][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:45:40,256][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:45:40,726][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:45:41,225][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:45:41,723][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:45:42,233][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:45:42,735][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:45:43,230][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:45:43,731][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:45:44,227][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:45:44,746][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:45:45,244][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:45:45,741][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:45:46,253][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:45:46,751][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:45:47,255][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:45:47,752][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:45:48,250][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:45:48,759][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:45:49,254][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:45:49,749][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:45:50,245][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:45:50,740][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:45:51,247][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:45:51,745][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:45:52,240][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:45:52,756][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:45:53,254][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:45:53,752][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:45:54,251][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:45:54,756][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:45:55,264][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:45:55,822][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:45:56,329][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:45:56,838][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:45:57,335][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:45:57,849][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:45:58,353][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:45:58,855][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:45:59,355][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:45:59,860][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:46:00,362][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:46:00,865][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:46:01,363][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:46:01,868][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:46:02,371][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:46:02,876][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:46:03,377][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:46:03,877][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:46:04,383][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:46:04,885][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:46:05,383][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:46:05,885][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:46:06,383][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:46:06,883][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:46:07,383][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:46:07,885][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:46:08,387][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:46:08,891][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:46:09,392][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:46:09,894][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:46:10,394][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:46:10,902][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:46:11,400][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:46:11,901][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:46:12,402][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10015 tokens.
[2025-11-13 07:46:13,256][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:33
[2025-11-13 07:46:14,014][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:46:14,016][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:46:14,018][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:46:16,058][__main__][INFO] - Iteration 601 took 1m 0s (38.04% Gen, 58.59% Train). Generation: 23s, Training: 35s. Estimated remaining time: 40h 55m 16s. Estimated total time: 50h 31m 50s. Time estimates for 10 more iterations: 10m 6s, 100 more iterations: 1h 41m 3s, 500 more iterations: 8h 25m 18s.
[2025-11-13 07:46:16,060][__main__][INFO] - Starting iteration 601.
[2025-11-13 07:46:16,536][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:46:16,537][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:46:37,449][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:46:37,535][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:46:48,838][__main__][INFO] - Number of regex retries in iteration 601: 2
[2025-11-13 07:46:48,839][__main__][INFO] - agents played in iteration 601 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:46:49,695][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:46:49,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:46:49,749][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:46:49,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:46:49,772][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:46:49,773][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:46:50,541][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:46:51,005][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:46:51,510][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:46:52,007][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:46:52,503][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:46:53,003][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:46:53,503][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:46:54,000][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:46:54,497][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:46:54,994][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:46:55,490][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:46:55,990][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:46:56,490][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:46:56,987][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:46:57,490][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:46:57,988][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:46:58,486][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:46:58,987][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:46:59,487][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:46:59,990][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:47:00,498][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:47:00,999][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:47:01,501][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:47:02,002][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:47:02,504][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:47:03,010][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:47:03,513][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:47:04,020][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:47:04,519][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:47:05,019][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:47:05,522][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:47:06,021][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:47:06,523][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:47:07,034][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:47:07,536][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:47:08,053][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:47:08,557][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:47:09,060][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:47:09,561][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:47:10,062][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:47:10,575][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:47:11,087][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:47:11,594][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:47:12,102][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:47:12,612][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:47:13,116][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:47:13,619][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:47:14,125][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:47:14,632][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:47:15,136][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:47:15,636][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:47:16,138][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:47:16,640][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:47:17,166][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:47:17,670][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:47:18,175][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:47:18,678][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:47:19,179][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:47:19,681][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:47:20,187][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:47:20,687][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:47:21,190][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:47:21,692][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:47:22,194][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:47:22,694][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10119 tokens.
[2025-11-13 07:47:23,599][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:33
[2025-11-13 07:47:24,297][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:47:24,298][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:47:24,301][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:47:25,286][__main__][INFO] - Iteration 602 took 1m 8s (46.98% Gen, 51.58% Train). Generation: 32s, Training: 35s. Estimated remaining time: 47h 39m 47s. Estimated total time: 57h 17m 30s. Time estimates for 10 more iterations: 11m 27s, 100 more iterations: 1h 54m 35s, 500 more iterations: 9h 32m 55s.
[2025-11-13 07:47:25,288][__main__][INFO] - Starting iteration 602.
[2025-11-13 07:47:25,775][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:47:25,776][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:47:50,515][__main__][INFO] - Number of regex retries in iteration 602: 0
[2025-11-13 07:47:50,516][__main__][INFO] - agents played in iteration 602 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:47:51,449][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:47:51,475][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:47:51,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:47:51,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:47:51,525][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:47:51,526][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:47:52,296][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:47:52,760][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:47:53,267][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:47:53,763][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:47:54,275][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:47:54,774][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:47:55,269][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:47:55,766][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:47:56,263][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:47:56,771][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:47:57,265][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:47:57,763][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:47:58,259][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:47:58,754][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:47:59,249][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:47:59,753][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:48:00,248][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:48:00,752][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:48:01,250][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:48:01,745][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:48:02,242][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:48:02,737][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:48:03,232][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:48:03,736][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:48:04,232][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:48:04,727][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:48:05,224][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:48:05,726][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:48:06,230][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:48:06,729][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:48:07,228][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:48:07,726][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:48:08,223][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:48:08,723][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:48:09,223][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:48:09,728][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:48:10,231][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:48:10,735][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:48:11,236][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:48:11,736][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:48:12,241][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:48:12,743][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:48:13,247][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:48:13,747][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:48:14,251][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:48:14,752][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:48:15,272][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:48:15,772][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:48:16,272][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:48:16,783][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:48:17,285][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:48:17,789][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:48:18,290][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:48:18,793][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:48:19,297][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:48:19,798][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:48:20,301][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:48:20,801][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:48:21,301][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:48:21,803][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:48:22,302][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:48:22,807][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:48:23,311][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:48:23,811][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:48:24,314][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9989 tokens.
[2025-11-13 07:48:25,233][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.01%, Current % of VRAM taken: 60.27%, Block Peak % of device VRAM: 61.96%, ΔTime: 00:00:32
[2025-11-13 07:48:25,988][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:48:25,991][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:48:25,993][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:48:27,001][__main__][INFO] - Iteration 603 took 1m 1s (40.41% Gen, 57.94% Train). Generation: 24s, Training: 35s. Estimated remaining time: 41h 22m 34s. Estimated total time: 51h 1m 19s. Time estimates for 10 more iterations: 10m 12s, 100 more iterations: 1h 42m 2s, 500 more iterations: 8h 30m 13s.
[2025-11-13 07:48:27,003][__main__][INFO] - Starting iteration 603.
[2025-11-13 07:48:27,482][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:48:27,483][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:48:51,992][__main__][INFO] - Number of regex retries in iteration 603: 0
[2025-11-13 07:48:51,993][__main__][INFO] - agents played in iteration 603 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:48:52,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:48:52,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:48:52,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:48:52,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.28%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:48:52,887][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:48:52,888][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:48:53,658][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:48:54,113][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:48:54,611][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:48:55,106][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:48:55,601][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:48:56,106][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:48:56,600][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:48:57,094][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:48:57,594][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:48:58,091][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:48:58,598][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:48:59,099][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:48:59,595][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:49:00,096][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:49:00,593][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:49:01,091][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:49:01,590][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:49:02,088][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:49:02,593][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:49:03,091][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:49:03,591][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:49:04,099][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:49:04,597][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:49:05,104][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:49:05,601][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:49:06,101][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:49:06,607][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:49:07,112][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:49:07,613][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:49:08,116][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:49:08,617][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:49:09,116][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:49:09,621][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:49:10,123][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:49:10,626][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:49:11,128][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:49:11,630][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:49:12,133][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:49:12,644][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:49:13,148][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:49:13,654][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:49:14,159][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:49:14,663][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:49:15,168][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:49:15,676][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:49:16,179][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:49:16,682][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:49:17,199][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:49:17,711][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:49:19,149][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:49:19,658][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:49:20,164][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:49:20,672][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:49:21,198][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:49:21,702][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:49:22,209][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:49:22,711][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:49:23,216][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:49:23,727][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:49:24,235][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:49:24,737][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:49:25,241][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:49:25,743][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:49:26,246][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:49:26,748][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10148 tokens.
[2025-11-13 07:49:27,712][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:34
[2025-11-13 07:49:28,446][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:49:28,448][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:49:28,450][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:49:29,370][__main__][INFO] - Iteration 604 took 1m 1s (39.60% Gen, 58.91% Train). Generation: 24s, Training: 36s. Estimated remaining time: 41h 54m 38s. Estimated total time: 51h 34m 25s. Time estimates for 10 more iterations: 10m 18s, 100 more iterations: 1h 43m 8s, 500 more iterations: 8h 35m 44s.
[2025-11-13 07:49:29,373][__main__][INFO] - Starting iteration 604.
[2025-11-13 07:49:29,849][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:49:29,850][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:49:58,686][__main__][INFO] - Number of regex retries in iteration 604: 0
[2025-11-13 07:49:58,686][__main__][INFO] - agents played in iteration 604 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:49:59,668][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:49:59,695][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:49:59,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:49:59,743][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:49:59,744][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:49:59,745][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:50:00,532][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:50:00,983][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:50:01,490][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:50:01,985][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:50:02,481][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:50:02,978][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:50:03,474][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:50:03,970][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:50:04,472][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:50:04,967][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:50:05,468][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:50:05,979][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:50:06,480][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:50:06,975][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:50:07,472][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:50:07,972][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:50:08,470][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:50:08,969][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:50:09,469][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:50:09,977][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:50:10,475][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:50:10,984][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:50:11,482][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:50:11,983][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:50:12,501][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:50:13,000][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:50:13,501][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:50:14,007][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:50:14,507][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:50:15,015][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:50:15,514][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:50:16,022][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:50:16,524][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:50:17,024][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:50:17,529][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:50:18,030][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:50:18,533][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:50:19,041][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:50:19,541][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:50:20,041][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:50:20,542][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:50:21,043][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:50:21,547][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:50:22,051][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:50:22,555][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:50:23,059][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:50:23,560][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:50:24,066][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:50:24,567][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:50:25,067][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:50:25,582][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:50:26,086][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:50:26,587][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:50:27,088][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:50:27,590][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:50:28,102][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:50:28,602][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:50:29,104][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:50:29,607][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:50:30,109][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:50:30,611][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:50:31,111][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:50:31,612][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:50:32,117][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:50:32,617][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10113 tokens.
[2025-11-13 07:50:33,572][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:33
[2025-11-13 07:50:34,354][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:50:34,356][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:50:34,358][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:50:35,408][__main__][INFO] - Iteration 605 took 1m 5s (43.99% Gen, 54.41% Train). Generation: 28s, Training: 35s. Estimated remaining time: 44h 57m 5s. Estimated total time: 54h 37m 58s. Time estimates for 10 more iterations: 10m 55s, 100 more iterations: 1h 49m 15s, 500 more iterations: 9h 6m 19s.
[2025-11-13 07:50:35,410][__main__][INFO] - Starting iteration 605.
[2025-11-13 07:50:35,886][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:50:35,887][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:51:04,918][__main__][INFO] - Number of regex retries in iteration 605: 0
[2025-11-13 07:51:04,918][__main__][INFO] - agents played in iteration 605 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:51:05,766][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:51:05,794][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:51:05,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:51:05,844][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:51:05,845][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:51:05,846][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:51:06,645][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:51:07,099][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:51:07,604][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:51:08,116][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:51:08,614][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:51:09,116][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:51:09,613][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:51:10,111][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:51:10,608][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:51:11,105][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:51:11,609][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:51:12,110][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:51:12,607][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:51:13,104][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:51:13,603][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:51:14,104][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:51:14,611][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:51:15,109][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:51:15,607][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:51:16,106][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:51:16,610][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:51:17,107][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:51:17,604][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:51:18,104][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:51:18,618][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:51:19,117][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:51:19,616][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:51:20,116][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:51:20,614][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:51:21,112][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:51:21,612][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:51:22,120][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:51:22,634][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:51:23,137][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:51:23,645][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:51:24,146][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:51:24,647][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:51:25,164][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:51:25,668][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:51:26,173][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:51:26,677][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:51:27,184][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:51:27,687][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:51:28,192][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:51:28,694][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:51:29,209][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:51:29,712][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:51:30,218][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:51:30,722][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:51:31,229][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:51:31,740][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:51:32,246][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:51:32,809][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:51:34,349][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:51:34,852][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:51:35,360][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:51:35,919][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:51:36,426][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:51:36,930][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:51:37,436][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:51:37,942][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:51:38,459][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:51:38,967][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:51:39,470][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:51:39,976][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10117 tokens.
[2025-11-13 07:51:40,854][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:34
[2025-11-13 07:51:41,501][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:51:41,503][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:51:41,505][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:51:42,446][__main__][INFO] - Iteration 606 took 1m 6s (43.62% Gen, 54.97% Train). Generation: 29s, Training: 36s. Estimated remaining time: 45h 46m 0s. Estimated total time: 55h 28m 1s. Time estimates for 10 more iterations: 11m 5s, 100 more iterations: 1h 50m 56s, 500 more iterations: 9h 14m 40s.
[2025-11-13 07:51:42,450][__main__][INFO] - Starting iteration 606.
[2025-11-13 07:51:42,938][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:51:42,939][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:52:01,424][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:52:12,480][__main__][INFO] - Number of regex retries in iteration 606: 1
[2025-11-13 07:52:12,481][__main__][INFO] - agents played in iteration 606 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:52:13,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:52:13,345][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:52:13,370][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:52:13,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:52:13,394][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:52:13,394][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:52:14,186][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:52:14,646][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:52:15,150][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:52:15,652][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:52:16,153][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:52:16,658][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:52:17,157][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:52:17,658][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:52:18,158][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:52:18,654][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:52:19,152][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:52:19,653][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:52:20,152][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:52:20,652][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:52:21,149][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:52:21,646][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:52:22,139][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:52:22,637][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:52:23,140][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:52:23,640][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:52:24,138][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:52:24,642][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:52:25,142][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:52:25,645][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:52:26,152][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:52:26,657][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:52:27,160][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:52:27,663][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:52:28,167][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:52:28,671][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:52:29,172][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:52:29,675][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:52:30,180][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:52:30,682][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:52:31,184][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:52:31,684][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:52:32,184][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:52:32,690][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:52:33,191][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:52:33,694][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:52:34,197][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:52:34,696][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:52:35,214][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:52:35,716][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:52:36,230][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:52:36,731][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:52:37,237][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:52:37,741][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:52:38,246][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:52:38,750][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:52:39,255][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:52:39,758][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:52:40,262][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:52:40,765][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:52:41,267][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:52:41,769][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:52:42,270][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:52:42,777][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:52:43,278][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:52:43,778][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:52:44,285][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:52:44,785][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:52:45,284][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:52:45,784][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:52:46,288][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10161 tokens.
[2025-11-13 07:52:47,195][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:33
[2025-11-13 07:52:47,984][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:52:47,986][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:52:47,987][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:52:49,090][__main__][INFO] - Iteration 607 took 1m 6s (44.66% Gen, 53.67% Train). Generation: 29s, Training: 35s. Estimated remaining time: 45h 24m 30s. Estimated total time: 55h 7m 37s. Time estimates for 10 more iterations: 11m 1s, 100 more iterations: 1h 50m 15s, 500 more iterations: 9h 11m 16s.
[2025-11-13 07:52:49,092][__main__][INFO] - Starting iteration 607.
[2025-11-13 07:52:49,575][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:52:49,576][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:53:07,095][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:53:16,408][__main__][INFO] - Number of regex retries in iteration 607: 1
[2025-11-13 07:53:16,409][__main__][INFO] - agents played in iteration 607 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:53:17,236][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:53:17,258][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:53:17,281][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:53:17,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:53:17,304][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:53:17,305][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:53:18,084][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:53:18,537][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:53:19,041][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:53:19,539][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:53:20,039][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:53:20,543][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:53:21,041][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:53:21,538][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:53:22,041][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:53:22,535][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:53:23,033][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:53:23,528][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:53:24,023][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:53:24,532][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:53:25,026][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:53:25,521][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:53:26,037][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:53:26,533][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:53:27,031][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:53:27,533][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:53:28,028][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:53:28,533][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:53:29,034][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:53:29,533][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:53:30,039][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:53:30,541][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:53:31,044][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:53:31,543][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:53:32,042][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:53:32,555][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:53:33,057][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:53:33,561][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:53:34,063][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:53:34,568][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:53:35,074][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:53:35,577][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:53:36,082][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:53:36,586][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:53:37,091][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:53:37,596][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:53:38,097][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:53:38,598][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:53:39,100][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:53:39,603][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:53:40,103][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:53:40,603][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:53:41,109][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:53:41,618][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:53:42,123][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:53:42,623][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:53:43,130][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:53:43,628][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:53:44,138][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:53:44,637][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:53:45,139][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:53:45,655][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:53:46,156][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:53:46,662][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:53:47,166][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:53:47,666][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:53:48,170][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:53:48,674][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:53:49,176][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:53:49,680][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:53:50,186][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10130 tokens.
[2025-11-13 07:53:51,122][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.08%, ΔTime: 00:00:33
[2025-11-13 07:53:51,890][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:53:51,891][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:53:51,893][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:53:53,204][__main__][INFO] - Iteration 608 took 1m 3s (42.17% Gen, 55.77% Train). Generation: 26s, Training: 35s. Estimated remaining time: 43h 17m 18s. Estimated total time: 53h 1m 30s. Time estimates for 10 more iterations: 10m 36s, 100 more iterations: 1h 46m 3s, 500 more iterations: 8h 50m 15s.
[2025-11-13 07:53:53,207][__main__][INFO] - Starting iteration 608.
[2025-11-13 07:53:53,696][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:53:53,697][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:54:20,751][__main__][INFO] - Number of regex retries in iteration 608: 0
[2025-11-13 07:54:20,751][__main__][INFO] - agents played in iteration 608 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:54:21,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:54:21,542][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:54:21,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:54:21,587][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:54:21,588][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:54:21,589][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:54:22,364][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:54:22,827][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:54:23,331][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:54:23,841][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:54:24,340][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:54:24,840][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:54:25,337][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:54:25,833][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:54:26,330][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:54:26,834][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:54:27,329][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:54:27,848][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:54:28,345][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:54:28,849][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:54:29,346][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:54:29,842][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:54:30,349][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:54:30,845][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:54:31,341][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:54:31,855][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:54:32,352][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:54:32,849][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:54:33,352][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:54:33,860][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:54:34,371][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:54:34,871][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:54:35,372][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:54:35,873][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:54:36,374][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:54:36,936][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:54:37,945][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:54:38,731][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:54:39,230][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:54:39,754][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:54:40,256][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:54:40,761][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:54:41,264][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:54:41,764][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:54:42,269][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:54:42,774][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:54:43,276][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:54:43,779][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:54:44,283][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:54:44,787][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:54:45,290][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:54:45,791][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:54:46,295][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:54:46,797][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:54:47,299][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:54:47,801][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:54:48,303][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:54:48,804][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:54:49,332][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:54:49,836][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:54:50,358][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:54:50,864][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:54:51,367][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:54:51,873][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:54:52,375][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:54:52,880][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:54:53,387][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:54:53,888][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:54:54,391][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:54:54,894][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:54:55,396][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10176 tokens.
[2025-11-13 07:54:56,319][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:33
[2025-11-13 07:54:57,009][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:54:57,011][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:54:57,014][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:54:58,219][__main__][INFO] - Iteration 609 took 1m 4s (41.93% Gen, 56.20% Train). Generation: 27s, Training: 36s. Estimated remaining time: 44h 0m 54s. Estimated total time: 53h 46m 10s. Time estimates for 10 more iterations: 10m 45s, 100 more iterations: 1h 47m 32s, 500 more iterations: 8h 57m 41s.
[2025-11-13 07:54:58,221][__main__][INFO] - Starting iteration 609.
[2025-11-13 07:54:58,704][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:54:58,705][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:55:27,866][__main__][INFO] - Number of regex retries in iteration 609: 0
[2025-11-13 07:55:27,867][__main__][INFO] - agents played in iteration 609 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:55:28,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:55:28,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:55:28,761][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:55:28,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:55:28,784][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:55:28,785][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:55:29,575][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:55:30,029][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:55:30,530][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:55:31,041][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:55:31,537][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:55:32,034][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:55:32,538][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:55:33,035][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:55:33,558][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:55:34,061][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:55:34,557][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:55:35,071][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:55:35,573][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:55:36,074][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:55:36,572][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:55:37,071][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:55:37,577][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:55:38,074][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:55:38,575][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:55:39,079][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:55:39,582][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:55:40,083][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:55:40,583][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:55:41,083][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:55:41,586][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:55:42,090][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:55:42,598][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:55:43,098][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:55:43,599][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:55:44,106][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:55:44,611][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:55:45,112][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:55:45,617][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:55:46,115][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:55:46,615][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:55:47,115][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:55:47,616][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:55:48,119][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:55:48,618][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:55:49,117][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:55:49,628][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:55:50,132][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:55:50,635][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:55:51,138][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:55:51,640][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:55:52,158][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:55:52,660][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:55:53,165][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:55:53,668][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:55:54,171][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:55:54,676][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:55:55,178][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:55:55,679][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:55:56,185][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:55:56,689][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:55:57,196][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:55:57,697][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:55:58,221][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:55:58,723][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:55:59,225][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:55:59,726][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:56:00,228][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:56:00,730][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:56:01,248][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:56:01,752][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10048 tokens.
[2025-11-13 07:56:02,719][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.13%, ΔTime: 00:00:33
[2025-11-13 07:56:03,456][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:56:03,458][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:56:03,460][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:56:04,543][__main__][INFO] - Iteration 610 took 1m 5s (44.29% Gen, 54.06% Train). Generation: 29s, Training: 35s. Estimated remaining time: 45h 5m 36s. Estimated total time: 54h 51m 59s. Time estimates for 10 more iterations: 10m 58s, 100 more iterations: 1h 49m 43s, 500 more iterations: 9h 8m 39s.
[2025-11-13 07:56:04,546][__main__][INFO] - Starting iteration 610.
[2025-11-13 07:56:05,037][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 07:56:05,037][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:56:30,532][__main__][INFO] - Number of regex retries in iteration 610: 0
[2025-11-13 07:56:30,533][__main__][INFO] - agents played in iteration 610 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:56:31,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:56:31,481][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:56:31,506][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:56:31,529][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:56:31,529][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:56:31,530][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:56:32,303][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:56:32,774][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:56:33,277][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:56:33,780][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:56:34,280][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:56:34,780][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:56:35,279][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:56:35,778][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:56:36,276][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:56:36,779][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:56:37,275][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:56:37,772][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:56:38,271][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:56:38,767][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:56:39,272][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:56:39,770][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:56:40,267][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:56:40,766][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:56:41,264][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:56:41,763][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:56:42,264][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:56:42,763][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:56:43,280][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:56:43,779][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:56:44,277][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:56:44,781][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:56:45,280][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:56:45,787][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:56:46,293][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:56:46,803][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:56:47,306][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:56:47,809][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:56:48,309][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:56:48,812][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:56:49,314][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:56:49,816][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:56:50,323][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:56:50,824][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:56:51,325][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:56:51,826][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:56:52,329][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:56:52,836][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:56:53,338][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:56:53,838][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:56:54,352][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:56:54,856][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:56:55,361][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:56:55,861][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:56:56,364][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:56:56,886][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:56:57,389][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:56:57,890][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:56:58,394][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:56:58,893][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:56:59,398][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:56:59,900][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:57:00,402][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:57:00,906][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:57:01,407][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:57:01,907][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:57:02,407][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:57:02,907][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:57:03,410][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:57:03,909][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:57:04,412][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10066 tokens.
[2025-11-13 07:57:05,262][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.33%, ΔTime: 00:00:32
[2025-11-13 07:57:06,040][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:57:06,041][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:57:06,043][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:57:08,005][__main__][INFO] - Iteration 611 took 1m 2s (40.49% Gen, 56.39% Train). Generation: 25s, Training: 35s. Estimated remaining time: 42h 41m 0s. Estimated total time: 52h 28m 26s. Time estimates for 10 more iterations: 10m 29s, 100 more iterations: 1h 44m 56s, 500 more iterations: 8h 44m 44s.
[2025-11-13 07:57:08,007][__main__][INFO] - Starting iteration 611.
[2025-11-13 07:57:08,508][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 07:57:08,510][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:57:21,071][mllm.models.large_language_model_local][WARNING] - Response Proposal: x hats, y books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:57:31,722][__main__][INFO] - Number of regex retries in iteration 611: 1
[2025-11-13 07:57:31,722][__main__][INFO] - agents played in iteration 611 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:57:32,491][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:57:32,515][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:57:32,540][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:57:32,562][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:57:32,563][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:57:32,564][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:57:33,339][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:57:33,797][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:57:34,298][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:57:34,793][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:57:35,296][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:57:35,796][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:57:36,299][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:57:36,797][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:57:37,292][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:57:37,791][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:57:38,290][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:57:38,787][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:57:39,284][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:57:39,781][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:57:40,293][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:57:40,793][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:57:41,290][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:57:41,799][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:57:42,295][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:57:42,792][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:57:43,289][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:57:43,791][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:57:44,306][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:57:44,806][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:57:45,308][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:57:45,811][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:57:46,997][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:57:47,493][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:57:47,992][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:57:48,496][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:57:48,995][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:57:49,504][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:57:50,003][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:57:50,502][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:57:51,009][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:57:51,509][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:57:52,014][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:57:52,514][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:57:53,017][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:57:53,522][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:57:54,028][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:57:54,529][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:57:55,030][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:57:55,535][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:57:56,038][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:57:56,543][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:57:57,048][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:57:57,555][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:57:58,055][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:57:58,560][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:57:59,074][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:57:59,577][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:58:00,081][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:58:00,585][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:58:01,090][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:58:01,612][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:58:02,116][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:58:02,621][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:58:03,122][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:58:03,622][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:58:04,124][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:58:04,626][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:58:05,128][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:58:05,635][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:58:06,136][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10052 tokens.
[2025-11-13 07:58:07,070][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.97%, Current % of VRAM taken: 60.22%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:33
[2025-11-13 07:58:07,717][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:58:07,719][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:58:07,721][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:58:08,659][__main__][INFO] - Iteration 612 took 1m 0s (38.59% Gen, 59.85% Train). Generation: 23s, Training: 36s. Estimated remaining time: 40h 19m 8s. Estimated total time: 50h 7m 35s. Time estimates for 10 more iterations: 10m 1s, 100 more iterations: 1h 40m 15s, 500 more iterations: 8h 21m 15s.
[2025-11-13 07:58:08,661][__main__][INFO] - Starting iteration 612.
[2025-11-13 07:58:09,145][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 07:58:09,145][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:58:39,233][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 07:58:40,147][__main__][INFO] - Number of regex retries in iteration 612: 1
[2025-11-13 07:58:40,148][__main__][INFO] - agents played in iteration 612 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:58:40,949][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.25%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:58:40,976][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.25%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:58:41,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.25%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:58:41,026][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.25%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:58:41,026][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:58:41,028][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:58:41,792][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:58:42,252][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:58:42,752][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:58:43,249][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:58:43,754][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:58:44,255][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:58:44,753][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:58:45,254][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:58:45,752][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:58:46,250][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:58:46,745][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:58:47,241][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:58:47,738][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:58:48,234][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:58:48,732][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:58:49,228][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:58:49,723][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:58:50,234][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:58:50,728][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:58:51,222][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:58:51,719][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:58:52,213][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:58:52,719][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:58:53,214][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:58:53,710][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:58:54,225][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:58:54,725][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:58:55,225][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:58:55,723][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:58:56,220][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:58:56,729][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:58:57,232][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:58:57,730][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:58:58,231][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 07:58:58,735][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 07:58:59,239][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 07:58:59,738][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 07:59:00,239][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 07:59:00,744][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 07:59:01,244][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 07:59:01,747][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 07:59:02,253][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 07:59:02,758][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 07:59:03,262][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 07:59:03,766][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 07:59:04,270][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 07:59:04,775][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 07:59:05,279][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 07:59:05,783][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 07:59:06,285][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 07:59:06,784][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 07:59:07,286][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 07:59:07,789][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 07:59:08,295][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 07:59:08,798][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 07:59:09,299][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 07:59:09,799][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 07:59:10,312][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 07:59:10,813][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 07:59:11,332][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 07:59:11,833][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 07:59:12,337][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 07:59:12,839][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 07:59:13,339][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 07:59:13,844][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10070 tokens.
[2025-11-13 07:59:14,757][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 07:59:15,513][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 07:59:15,515][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 07:59:15,517][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 07:59:16,770][__main__][INFO] - Iteration 613 took 1m 7s (45.84% Gen, 52.30% Train). Generation: 31s, Training: 35s. Estimated remaining time: 46h 31m 43s. Estimated total time: 56h 21m 18s. Time estimates for 10 more iterations: 11m 16s, 100 more iterations: 1h 52m 42s, 500 more iterations: 9h 23m 33s.
[2025-11-13 07:59:16,772][__main__][INFO] - Starting iteration 613.
[2025-11-13 07:59:17,251][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 07:59:17,252][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 07:59:41,864][__main__][INFO] - Number of regex retries in iteration 613: 0
[2025-11-13 07:59:41,864][__main__][INFO] - agents played in iteration 613 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 07:59:42,627][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:59:42,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:59:42,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:59:42,699][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 07:59:42,699][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 07:59:42,700][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 07:59:43,475][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 07:59:43,927][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 07:59:44,441][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 07:59:44,937][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 07:59:45,442][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 07:59:45,938][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 07:59:46,434][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 07:59:46,931][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 07:59:47,428][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 07:59:47,924][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 07:59:48,436][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 07:59:48,937][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 07:59:49,435][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 07:59:49,953][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 07:59:50,454][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 07:59:50,957][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 07:59:51,453][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 07:59:51,952][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 07:59:52,454][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 07:59:52,950][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 07:59:53,446][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 07:59:53,953][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 07:59:54,449][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 07:59:54,943][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 07:59:55,438][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 07:59:55,934][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 07:59:56,431][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 07:59:56,928][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 07:59:57,423][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 07:59:57,938][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 07:59:58,433][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 07:59:58,929][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 07:59:59,431][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 07:59:59,937][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:00:00,457][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:00:00,958][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:00:01,462][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:00:01,961][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:00:02,464][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:00:02,969][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:00:03,471][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:00:03,974][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:00:04,475][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:00:04,979][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:00:05,481][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:00:05,982][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:00:06,481][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:00:06,982][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:00:07,481][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:00:07,981][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:00:08,482][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:00:08,983][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:00:09,483][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:00:09,983][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:00:10,485][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:00:10,987][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:00:11,488][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:00:11,991][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:00:12,495][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:00:12,997][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:00:13,502][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:00:14,006][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:00:14,509][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:00:15,011][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:00:15,516][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10071 tokens.
[2025-11-13 08:00:16,353][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:32
[2025-11-13 08:00:17,080][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:00:17,082][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:00:17,084][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:00:18,249][__main__][INFO] - Iteration 614 took 1m 0s (40.35% Gen, 57.74% Train). Generation: 24s, Training: 35s. Estimated remaining time: 40h 59m 19s. Estimated total time: 50h 49m 55s. Time estimates for 10 more iterations: 10m 9s, 100 more iterations: 1h 41m 39s, 500 more iterations: 8h 28m 19s.
[2025-11-13 08:00:18,251][__main__][INFO] - Starting iteration 614.
[2025-11-13 08:00:18,721][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 08:00:18,721][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:00:39,347][mllm.models.large_language_model_local][WARNING] - Response Proposal: x hats, y books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:00:51,032][__main__][INFO] - Number of regex retries in iteration 614: 1
[2025-11-13 08:00:51,033][__main__][INFO] - agents played in iteration 614 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:00:51,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:00:51,894][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:00:51,919][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:00:51,942][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:00:51,943][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:00:51,943][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:00:52,720][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:00:53,175][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:00:53,674][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:00:54,176][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:00:54,674][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:00:55,171][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:00:55,671][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:00:56,170][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:00:56,665][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:00:57,167][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:00:57,671][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:00:58,179][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:00:58,686][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:00:59,192][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:00:59,699][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:01:00,204][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:01:00,704][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:01:01,207][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:01:01,709][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:01:02,212][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:01:02,714][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:01:03,216][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:01:03,723][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:01:04,226][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:01:04,731][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:01:05,235][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:01:05,741][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:01:06,245][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:01:06,748][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:01:07,265][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:01:07,766][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:01:08,273][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:01:08,772][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:01:09,273][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:01:09,775][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:01:10,277][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:01:10,778][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:01:11,278][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:01:11,780][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:01:12,284][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:01:12,784][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:01:13,287][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:01:13,790][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:01:14,291][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:01:14,794][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:01:15,296][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:01:15,799][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:01:16,302][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:01:16,803][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:01:17,308][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:01:17,811][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:01:18,316][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:01:18,821][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:01:19,324][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:01:19,828][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:01:20,334][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:01:20,839][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:01:21,338][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:01:21,839][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:01:22,339][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:01:22,839][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:01:23,341][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:01:23,842][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:01:24,340][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:01:24,840][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10193 tokens.
[2025-11-13 08:01:25,762][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.97%, Current % of VRAM taken: 60.22%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 08:01:26,403][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:01:26,405][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:01:26,407][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:01:27,352][__main__][INFO] - Iteration 615 took 1m 8s (47.08% Gen, 51.54% Train). Generation: 32s, Training: 35s. Estimated remaining time: 47h 19m 50s. Estimated total time: 57h 11m 36s. Time estimates for 10 more iterations: 11m 26s, 100 more iterations: 1h 54m 23s, 500 more iterations: 9h 31m 56s.
[2025-11-13 08:01:27,355][__main__][INFO] - Starting iteration 615.
[2025-11-13 08:01:27,837][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 08:01:27,839][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:01:53,777][__main__][INFO] - Number of regex retries in iteration 615: 0
[2025-11-13 08:01:53,777][__main__][INFO] - agents played in iteration 615 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:01:54,613][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:01:54,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:01:54,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:01:54,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.26%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:01:54,685][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:01:54,686][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:01:55,470][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:01:56,060][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:01:56,565][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:01:57,066][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:01:57,563][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:01:58,059][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:01:58,557][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:01:59,056][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:01:59,557][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:02:00,059][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:02:00,556][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:02:01,062][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:02:01,558][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:02:02,056][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:02:02,561][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:02:03,064][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:02:03,558][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:02:04,056][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:02:04,551][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:02:05,047][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:02:05,546][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:02:06,040][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:02:06,552][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:02:07,048][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:02:07,545][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:02:08,042][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:02:08,541][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:02:09,037][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:02:09,532][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:02:10,027][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:02:10,523][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:02:11,019][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:02:11,514][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:02:12,009][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:02:12,504][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:02:13,000][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:02:13,496][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:02:13,993][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:02:14,495][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:02:14,995][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:02:15,497][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:02:16,000][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:02:16,502][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:02:17,005][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:02:17,504][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:02:18,005][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:02:18,509][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:02:19,008][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:02:19,509][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:02:20,013][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:02:20,515][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:02:21,019][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:02:21,519][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:02:22,019][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:02:22,520][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:02:23,021][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:02:23,526][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:02:24,025][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:02:24,526][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:02:25,038][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:02:25,540][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:02:26,041][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:02:26,540][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:02:27,040][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:02:27,556][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9988 tokens.
[2025-11-13 08:02:28,469][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 61.93%, ΔTime: 00:00:33
[2025-11-13 08:02:29,253][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:02:29,255][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:02:29,257][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:02:30,404][__main__][INFO] - Iteration 616 took 1m 2s (41.45% Gen, 56.71% Train). Generation: 25s, Training: 35s. Estimated remaining time: 42h 15m 32s. Estimated total time: 52h 8m 21s. Time estimates for 10 more iterations: 10m 25s, 100 more iterations: 1h 44m 16s, 500 more iterations: 8h 41m 23s.
[2025-11-13 08:02:30,406][__main__][INFO] - Starting iteration 616.
[2025-11-13 08:02:30,887][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 08:02:30,888][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:02:55,354][__main__][INFO] - Number of regex retries in iteration 616: 0
[2025-11-13 08:02:55,355][__main__][INFO] - agents played in iteration 616 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:02:56,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:02:56,298][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:02:56,324][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:02:56,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:02:56,347][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:02:56,348][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:02:57,122][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:02:57,579][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:02:58,087][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:02:58,582][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:02:59,079][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:02:59,579][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:03:00,075][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:03:00,572][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:03:01,072][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:03:01,566][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:03:02,062][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:03:02,558][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:03:03,060][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:03:03,555][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:03:04,052][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:03:04,555][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:03:05,052][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:03:05,548][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:03:06,045][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:03:06,542][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:03:07,051][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:03:07,548][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:03:08,050][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:03:08,546][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:03:09,042][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:03:09,540][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:03:10,035][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:03:10,532][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:03:11,029][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:03:11,533][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:03:12,030][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:03:12,525][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:03:13,020][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:03:13,516][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:03:14,011][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:03:14,507][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:03:15,001][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:03:15,496][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:03:15,992][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:03:16,489][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:03:16,988][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:03:17,491][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:03:17,990][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:03:18,492][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:03:18,993][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:03:19,492][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:03:19,996][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:03:20,502][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:03:21,009][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:03:21,507][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:03:22,006][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:03:22,509][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:03:23,013][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:03:23,513][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:03:24,056][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:03:25,410][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:03:25,913][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:03:26,414][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:03:26,922][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:03:27,424][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:03:27,928][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:03:28,428][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:03:28,929][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:03:29,431][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:03:29,933][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10039 tokens.
[2025-11-13 08:03:30,873][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:33
[2025-11-13 08:03:31,547][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:03:31,549][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:03:31,552][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:03:32,462][__main__][INFO] - Iteration 617 took 1m 1s (39.73% Gen, 58.78% Train). Generation: 24s, Training: 36s. Estimated remaining time: 41h 24m 57s. Estimated total time: 51h 18m 48s. Time estimates for 10 more iterations: 10m 15s, 100 more iterations: 1h 42m 37s, 500 more iterations: 8h 33m 8s.
[2025-11-13 08:03:32,466][__main__][INFO] - Starting iteration 617.
[2025-11-13 08:03:32,957][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 08:03:32,958][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:04:07,333][__main__][INFO] - Number of regex retries in iteration 617: 0
[2025-11-13 08:04:07,333][__main__][INFO] - agents played in iteration 617 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:04:08,161][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:04:08,188][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:04:08,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:04:08,236][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:04:08,236][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:04:08,238][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:04:09,036][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:04:09,494][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:04:10,008][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:04:10,506][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:04:11,005][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:04:11,508][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:04:12,012][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:04:12,516][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:04:13,014][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:04:13,512][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:04:14,018][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:04:14,517][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:04:15,017][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:04:15,513][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:04:16,013][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:04:16,516][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:04:17,014][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:04:17,515][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:04:18,018][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:04:18,515][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:04:19,011][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:04:19,512][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:04:20,011][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:04:20,519][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:04:21,021][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:04:21,524][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:04:22,029][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:04:22,533][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:04:23,038][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:04:23,540][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:04:24,040][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:04:24,542][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:04:25,047][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:04:25,549][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:04:26,054][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:04:26,558][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:04:27,064][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:04:27,574][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:04:28,079][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:04:28,597][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:04:29,102][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:04:29,603][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:04:30,106][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:04:30,607][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:04:31,117][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:04:31,617][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:04:32,121][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:04:32,625][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:04:33,125][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:04:33,627][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:04:34,125][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:04:34,634][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:04:35,140][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:04:35,641][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:04:36,144][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:04:36,648][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:04:37,154][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:04:37,664][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:04:38,169][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:04:38,676][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:04:39,213][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:04:39,718][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:04:40,222][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:04:40,725][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:04:41,231][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10165 tokens.
[2025-11-13 08:04:42,069][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.29%, Current % of VRAM taken: 60.55%, Block Peak % of device VRAM: 62.44%, ΔTime: 00:00:33
[2025-11-13 08:04:42,860][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:04:42,862][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:04:42,863][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:04:43,873][__main__][INFO] - Iteration 618 took 1m 10s (48.47% Gen, 50.10% Train). Generation: 34s, Training: 35s. Estimated remaining time: 49h 10m 48s. Estimated total time: 59h 5m 50s. Time estimates for 10 more iterations: 11m 49s, 100 more iterations: 1h 58m 11s, 500 more iterations: 9h 50m 58s.
[2025-11-13 08:04:43,876][__main__][INFO] - Starting iteration 618.
[2025-11-13 08:04:44,364][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 08:04:44,365][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:05:11,848][__main__][INFO] - Number of regex retries in iteration 618: 0
[2025-11-13 08:05:11,848][__main__][INFO] - agents played in iteration 618 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:05:12,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:05:12,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:05:12,688][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:05:12,713][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:05:12,713][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:05:12,714][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:05:13,486][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:05:13,941][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:05:14,444][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:05:14,940][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:05:15,438][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:05:15,939][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:05:16,441][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:05:16,937][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:05:17,436][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:05:17,932][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:05:18,426][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:05:18,925][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:05:19,427][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:05:19,922][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:05:20,417][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:05:20,916][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:05:21,411][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:05:21,907][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:05:22,403][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:05:22,899][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:05:23,413][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:05:23,909][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:05:24,405][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:05:24,902][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:05:25,402][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:05:25,912][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:05:26,412][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:05:26,913][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:05:27,426][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:05:27,924][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:05:28,429][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:05:28,931][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:05:29,432][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:05:29,936][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:05:30,435][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:05:30,937][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:05:31,440][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:05:31,941][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:05:32,448][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:05:32,950][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:05:33,451][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:05:33,958][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:05:34,463][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:05:34,968][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:05:35,479][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:05:35,984][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:05:36,499][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:05:36,999][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:05:37,503][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:05:38,010][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:05:38,514][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:05:39,016][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:05:39,521][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:05:40,024][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:05:40,535][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:05:41,039][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:05:41,544][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:05:42,045][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:05:42,545][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:05:43,048][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:05:43,549][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:05:44,051][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:05:44,555][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:05:45,193][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:05:46,309][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10175 tokens.
[2025-11-13 08:05:47,460][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:33
[2025-11-13 08:05:48,536][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:05:48,537][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:05:48,539][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:05:49,496][__main__][INFO] - Iteration 619 took 1m 5s (42.20% Gen, 56.33% Train). Generation: 27s, Training: 36s. Estimated remaining time: 44h 20m 30s. Estimated total time: 54h 16m 38s. Time estimates for 10 more iterations: 10m 51s, 100 more iterations: 1h 48m 33s, 500 more iterations: 9h 2m 46s.
[2025-11-13 08:05:49,499][__main__][INFO] - Starting iteration 619.
[2025-11-13 08:05:49,981][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 08:05:49,982][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:06:24,278][__main__][INFO] - Number of regex retries in iteration 619: 0
[2025-11-13 08:06:24,279][__main__][INFO] - agents played in iteration 619 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:06:25,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:25,152][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:25,176][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:25,199][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:25,200][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:06:25,200][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:06:26,058][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:06:26,525][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:06:27,033][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:06:27,536][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:06:28,037][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:06:28,537][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:06:29,040][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:06:29,540][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:06:30,046][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:06:30,550][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:06:31,056][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:06:31,558][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:06:32,065][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:06:32,565][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:06:33,069][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:06:33,574][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:06:34,073][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:06:34,572][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:06:35,074][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:06:35,577][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:06:36,078][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:06:36,579][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:06:37,084][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:06:37,583][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:06:38,090][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:06:38,589][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:06:39,092][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:06:39,597][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:06:40,097][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:06:40,597][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:06:41,096][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:06:41,595][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:06:42,113][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:06:42,622][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:06:43,123][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:06:43,623][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:06:44,126][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:06:44,634][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:06:45,136][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:06:45,640][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:06:46,147][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:06:46,647][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:06:47,147][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:06:47,646][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:06:48,146][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:06:48,649][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:06:49,149][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:06:49,650][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:06:50,150][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:06:50,656][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:06:51,156][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:06:51,661][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:06:52,163][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:06:52,667][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:06:53,169][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:06:53,668][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:06:54,170][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:06:54,671][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:06:55,175][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:06:55,676][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:06:56,177][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:06:56,683][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:06:57,182][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:06:57,684][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:06:58,185][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10026 tokens.
[2025-11-13 08:06:59,096][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:33
[2025-11-13 08:06:59,872][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:06:59,873][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:06:59,875][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:07:00,911][__main__][INFO] - Iteration 620 took 1m 10s (48.35% Gen, 50.19% Train). Generation: 34s, Training: 35s. Estimated remaining time: 49h 9m 10s. Estimated total time: 59h 6m 29s. Time estimates for 10 more iterations: 11m 49s, 100 more iterations: 1h 58m 12s, 500 more iterations: 9h 51m 4s.
[2025-11-13 08:07:00,913][__main__][INFO] - Starting iteration 620.
[2025-11-13 08:07:01,390][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 08:07:01,391][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:07:26,407][__main__][INFO] - Number of regex retries in iteration 620: 0
[2025-11-13 08:07:26,407][__main__][INFO] - agents played in iteration 620 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:07:27,318][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:27,345][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:27,371][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:27,394][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:27,394][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:07:27,395][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:07:28,184][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:07:28,638][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:07:29,147][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:07:29,643][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:07:30,139][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:07:30,638][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:07:31,142][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:07:31,641][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:07:32,153][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:07:32,655][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:07:33,161][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:07:33,660][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:07:34,162][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:07:34,664][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:07:35,167][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:07:35,672][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:07:36,177][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:07:36,680][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:07:37,183][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:07:37,685][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:07:38,190][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:07:38,691][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:07:39,192][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:07:39,696][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:07:40,199][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:07:40,699][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:07:41,197][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:07:41,699][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:07:42,198][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:07:42,699][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:07:43,198][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:07:43,697][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:07:44,204][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:07:44,704][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:07:45,214][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:07:45,717][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:07:46,219][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:07:46,739][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:07:47,240][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:07:47,742][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:07:48,247][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:07:48,751][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:07:49,258][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:07:49,762][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:07:50,262][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:07:50,765][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:07:51,268][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:07:51,770][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:07:52,269][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:07:52,770][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:07:53,273][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:07:53,774][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:07:54,274][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:07:54,779][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:07:55,279][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:07:55,781][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:07:56,284][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:07:56,789][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:07:57,292][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:07:57,796][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:07:58,300][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:07:58,804][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:07:59,307][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:07:59,820][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:08:00,325][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10130 tokens.
[2025-11-13 08:08:01,234][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:33
[2025-11-13 08:08:02,020][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:08:02,021][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:08:02,023][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:08:04,100][__main__][INFO] - Iteration 621 took 1m 2s (39.89% Gen, 56.80% Train). Generation: 25s, Training: 35s. Estimated remaining time: 42h 17m 7s. Estimated total time: 52h 15m 29s. Time estimates for 10 more iterations: 10m 27s, 100 more iterations: 1h 44m 30s, 500 more iterations: 8h 42m 34s.
[2025-11-13 08:08:04,102][__main__][INFO] - Starting iteration 621.
[2025-11-13 08:08:04,586][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:08:04,587][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:08:16,231][mllm.models.large_language_model_local][WARNING] - Response Proposal: x hats, y books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:08:27,161][__main__][INFO] - Number of regex retries in iteration 621: 1
[2025-11-13 08:08:27,161][__main__][INFO] - agents played in iteration 621 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:08:27,935][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:27,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:27,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:28,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:28,013][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:08:28,014][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:08:28,805][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:08:29,261][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:08:29,765][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:08:30,263][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:08:30,759][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:08:31,260][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:08:31,755][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:08:32,250][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:08:32,745][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:08:33,240][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:08:33,746][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:08:34,249][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:08:34,749][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:08:35,261][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:08:35,764][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:08:36,268][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:08:36,773][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:08:37,271][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:08:37,770][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:08:38,272][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:08:38,772][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:08:39,274][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:08:39,774][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:08:40,278][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:08:40,778][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:08:41,282][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:08:41,785][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:08:42,285][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:08:42,787][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:08:43,289][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:08:43,795][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:08:44,301][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:08:44,807][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:08:45,306][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:08:45,815][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:08:46,315][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:08:47,843][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:08:48,345][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:08:48,852][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:08:49,352][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:08:49,852][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:08:50,357][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:08:50,864][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:08:51,369][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:08:51,871][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:08:52,380][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:08:52,881][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:08:53,381][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:08:53,882][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:08:54,385][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:08:54,891][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:08:55,393][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:08:55,896][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:08:56,399][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:08:56,905][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:08:57,411][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:08:57,921][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:08:58,420][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:08:58,937][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:08:59,437][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:08:59,949][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:09:00,449][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:09:00,949][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:09:01,461][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:09:01,962][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10201 tokens.
[2025-11-13 08:09:02,889][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:34
[2025-11-13 08:09:03,529][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:09:03,531][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:09:03,532][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:09:04,483][__main__][INFO] - Iteration 622 took 59s (37.69% Gen, 60.72% Train). Generation: 22s, Training: 36s. Estimated remaining time: 39h 55m 30s. Estimated total time: 49h 54m 53s. Time estimates for 10 more iterations: 9m 58s, 100 more iterations: 1h 39m 49s, 500 more iterations: 8h 19m 8s.
[2025-11-13 08:09:04,486][__main__][INFO] - Starting iteration 622.
[2025-11-13 08:09:04,998][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:09:04,998][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:09:29,460][__main__][INFO] - Number of regex retries in iteration 622: 0
[2025-11-13 08:09:29,461][__main__][INFO] - agents played in iteration 622 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:09:30,400][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:30,427][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:30,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:30,476][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:30,477][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:09:30,477][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:09:31,285][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:09:31,738][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:09:32,238][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:09:32,743][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:09:33,244][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:09:33,746][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:09:34,242][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:09:34,739][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:09:35,241][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:09:35,863][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:09:36,374][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:09:36,872][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:09:37,370][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:09:37,869][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:09:38,366][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:09:38,872][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:09:39,396][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:09:39,896][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:09:40,398][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:09:40,902][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:09:41,401][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:09:41,899][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:09:42,399][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:09:42,901][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:09:43,401][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:09:43,904][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:09:44,407][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:09:44,911][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:09:45,418][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:09:45,922][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:09:46,424][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:09:46,929][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:09:47,435][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:09:47,936][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:09:48,453][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:09:48,950][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:09:49,465][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:09:49,966][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:09:50,465][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:09:50,968][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:09:51,471][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:09:51,977][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:09:52,478][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:09:52,979][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:09:53,484][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:09:53,984][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:09:54,490][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:09:54,989][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:09:55,490][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:09:55,994][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:09:56,495][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:09:56,997][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:09:57,498][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:09:57,999][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:09:58,501][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:09:59,003][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:09:59,503][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:10:00,005][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:10:00,508][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:10:01,003][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:10:01,499][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:10:01,998][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:10:02,499][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:10:02,997][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:10:03,497][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10127 tokens.
[2025-11-13 08:10:04,391][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:33
[2025-11-13 08:10:05,182][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:10:05,184][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:10:05,186][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:10:06,250][__main__][INFO] - Iteration 623 took 1m 1s (39.94% Gen, 58.32% Train). Generation: 24s, Training: 35s. Estimated remaining time: 41h 2m 14s. Estimated total time: 51h 2m 38s. Time estimates for 10 more iterations: 10m 12s, 100 more iterations: 1h 42m 5s, 500 more iterations: 8h 30m 26s.
[2025-11-13 08:10:06,253][__main__][INFO] - Starting iteration 623.
[2025-11-13 08:10:06,738][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:10:06,739][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:10:34,742][__main__][INFO] - Number of regex retries in iteration 623: 0
[2025-11-13 08:10:34,743][__main__][INFO] - agents played in iteration 623 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:10:35,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:35,682][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:35,707][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:35,729][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:35,730][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:10:35,731][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:10:36,509][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:10:36,960][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:10:37,462][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:10:37,984][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:10:38,480][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:10:38,975][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:10:39,475][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:10:39,971][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:10:40,468][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:10:40,962][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:10:41,458][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:10:41,986][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:10:42,480][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:10:42,975][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:10:43,474][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:10:43,970][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:10:44,474][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:10:44,976][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:10:45,477][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:10:45,989][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:10:46,488][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:10:46,988][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:10:47,499][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:10:47,998][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:10:48,502][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:10:49,001][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:10:49,506][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:10:50,009][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:10:50,509][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:10:51,009][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:10:51,509][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:10:52,013][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:10:52,518][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:10:53,020][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:10:53,521][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:10:54,021][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:10:54,521][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:10:55,027][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:10:55,529][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:10:56,028][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:10:56,531][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:10:57,031][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:10:57,530][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:10:58,034][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:10:58,533][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:10:59,035][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:10:59,536][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:11:00,039][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:11:00,536][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:11:01,032][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:11:01,528][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:11:02,025][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:11:02,520][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:11:03,016][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:11:03,511][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:11:04,006][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:11:04,506][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:11:05,002][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:11:05,501][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:11:06,002][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:11:06,502][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:11:07,005][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:11:07,504][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:11:08,002][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:11:08,502][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9984 tokens.
[2025-11-13 08:11:09,418][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:32
[2025-11-13 08:11:10,181][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:11:10,183][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:11:10,185][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:11:11,208][__main__][INFO] - Iteration 624 took 1m 4s (43.44% Gen, 54.98% Train). Generation: 28s, Training: 35s. Estimated remaining time: 43h 42m 0s. Estimated total time: 53h 43m 29s. Time estimates for 10 more iterations: 10m 44s, 100 more iterations: 1h 47m 26s, 500 more iterations: 8h 57m 14s.
[2025-11-13 08:11:11,210][__main__][INFO] - Starting iteration 624.
[2025-11-13 08:11:11,700][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:11:11,701][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:11:35,647][__main__][INFO] - Number of regex retries in iteration 624: 0
[2025-11-13 08:11:35,648][__main__][INFO] - agents played in iteration 624 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:11:36,511][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:36,543][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:36,571][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:36,594][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:36,595][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:11:36,596][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:11:37,392][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:11:37,847][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:11:38,349][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:11:38,845][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:11:39,341][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:11:39,839][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:11:40,336][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:11:40,833][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:11:41,332][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:11:41,828][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:11:42,325][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:11:42,821][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:11:43,316][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:11:43,815][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:11:44,312][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:11:44,813][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:11:45,308][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:11:45,810][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:11:46,316][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:11:46,816][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:11:47,320][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:11:47,822][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:11:48,321][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:11:48,822][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:11:49,318][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:11:49,825][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:11:50,328][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:11:50,831][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:11:51,336][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:11:51,843][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:11:52,346][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:11:52,848][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:11:53,353][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:11:53,854][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:11:54,363][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:11:54,867][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:11:55,366][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:11:55,874][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:11:56,375][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:11:56,887][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:11:57,387][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:11:57,891][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:11:58,407][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:11:58,909][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:11:59,412][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:11:59,913][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:12:00,415][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:12:00,919][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:12:01,422][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:12:01,924][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:12:02,430][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:12:02,932][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:12:03,437][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:12:03,939][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:12:04,441][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:12:04,947][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:12:05,452][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:12:05,951][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:12:06,451][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:12:06,948][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:12:07,455][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:12:07,955][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:12:08,458][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:12:08,972][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:12:09,476][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10062 tokens.
[2025-11-13 08:12:10,406][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:33
[2025-11-13 08:12:11,092][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:12:11,094][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:12:11,096][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:12:12,574][__main__][INFO] - Iteration 625 took 1m 0s (39.34% Gen, 58.23% Train). Generation: 23s, Training: 35s. Estimated remaining time: 40h 41m 12s. Estimated total time: 50h 43m 43s. Time estimates for 10 more iterations: 10m 8s, 100 more iterations: 1h 41m 27s, 500 more iterations: 8h 27m 17s.
[2025-11-13 08:12:12,576][__main__][INFO] - Starting iteration 625.
[2025-11-13 08:12:13,055][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:12:13,055][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:12:32,168][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:12:32,736][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:12:36,467][__main__][INFO] - Number of regex retries in iteration 625: 2
[2025-11-13 08:12:36,467][__main__][INFO] - agents played in iteration 625 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:12:37,388][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:37,411][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:37,437][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:37,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:37,460][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:12:37,461][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:12:38,356][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:12:38,814][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:12:39,314][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:12:39,811][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:12:40,313][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:12:40,808][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:12:41,304][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:12:41,807][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:12:42,302][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:12:42,797][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:12:43,293][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:12:43,791][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:12:44,285][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:12:44,787][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:12:45,281][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:12:45,792][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:12:46,288][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:12:46,787][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:12:47,284][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:12:47,780][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:12:48,291][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:12:48,792][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:12:49,289][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:12:49,790][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:12:50,284][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:12:50,780][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:12:51,280][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:12:51,776][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:12:52,295][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:12:52,792][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:12:53,287][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:12:53,786][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:12:54,281][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:12:54,779][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:12:55,280][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:12:55,775][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:12:56,290][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:12:56,786][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:12:57,281][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:12:57,783][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:12:58,284][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:12:58,793][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:12:59,291][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:12:59,791][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:13:00,303][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:13:00,800][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:13:01,300][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:13:01,805][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:13:02,304][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:13:02,804][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:13:03,303][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:13:03,803][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:13:04,310][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:13:04,814][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:13:05,317][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:13:05,819][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:13:06,315][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:13:06,819][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:13:07,329][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:13:07,833][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:13:08,337][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:13:08,841][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:13:09,346][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:13:09,847][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:13:10,349][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10125 tokens.
[2025-11-13 08:13:11,281][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.31%, ΔTime: 00:00:32
[2025-11-13 08:13:12,039][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:13:12,041][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:13:12,043][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:13:13,061][__main__][INFO] - Iteration 626 took 1m 0s (39.01% Gen, 59.29% Train). Generation: 23s, Training: 35s. Estimated remaining time: 39h 56m 49s. Estimated total time: 50h 0m 21s. Time estimates for 10 more iterations: 10m 0s, 100 more iterations: 1h 40m 0s, 500 more iterations: 8h 20m 3s.
[2025-11-13 08:13:13,063][__main__][INFO] - Starting iteration 626.
[2025-11-13 08:13:13,609][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:13:13,610][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:13:31,240][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:13:37,154][__main__][INFO] - Number of regex retries in iteration 626: 1
[2025-11-13 08:13:37,155][__main__][INFO] - agents played in iteration 626 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:13:38,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:38,118][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:38,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:38,165][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:38,165][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:13:38,167][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:13:38,987][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:13:39,440][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:13:39,949][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:13:40,445][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:13:40,944][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:13:41,439][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:13:41,936][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:13:42,432][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:13:42,929][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:13:43,430][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:13:43,926][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:13:44,421][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:13:44,919][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:13:45,417][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:13:45,913][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:13:46,407][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:13:46,901][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:13:47,405][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:13:47,899][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:13:48,396][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:13:48,892][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:13:49,385][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:13:49,895][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:13:50,389][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:13:50,883][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:13:51,380][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:13:51,875][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:13:52,371][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:13:52,875][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:13:53,372][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:13:53,872][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:13:54,372][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:13:54,866][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:13:55,361][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:13:55,856][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:13:56,352][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:13:56,846][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:13:57,340][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:13:57,837][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:13:58,332][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:13:58,828][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:13:59,325][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:13:59,824][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:14:00,327][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:14:00,830][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:14:01,335][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:14:01,836][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:14:02,332][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:14:02,831][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:14:03,331][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:14:03,926][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:14:04,470][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:14:04,974][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:14:05,477][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:14:05,984][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:14:06,490][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:14:07,005][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:14:07,509][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:14:08,008][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:14:08,508][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:14:09,009][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:14:09,509][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:14:10,007][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:14:10,510][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:14:11,019][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10024 tokens.
[2025-11-13 08:14:11,978][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 62.03%, ΔTime: 00:00:32
[2025-11-13 08:14:12,660][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:14:12,662][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:14:12,664][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:14:13,591][__main__][INFO] - Iteration 627 took 59s (39.25% Gen, 59.20% Train). Generation: 23s, Training: 35s. Estimated remaining time: 39h 54m 35s. Estimated total time: 49h 59m 7s. Time estimates for 10 more iterations: 9m 59s, 100 more iterations: 1h 39m 58s, 500 more iterations: 8h 19m 51s.
[2025-11-13 08:14:13,595][__main__][INFO] - Starting iteration 627.
[2025-11-13 08:14:14,137][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:14:14,137][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:14:43,072][__main__][INFO] - Number of regex retries in iteration 627: 0
[2025-11-13 08:14:43,073][__main__][INFO] - agents played in iteration 627 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:14:43,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:43,971][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:43,996][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:44,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:44,020][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:14:44,021][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:14:44,873][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:14:45,334][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:14:45,835][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:14:46,336][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:14:46,839][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:14:47,339][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:14:47,837][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:14:48,333][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:14:48,832][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:14:49,330][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:14:49,826][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:14:50,324][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:14:50,820][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:14:51,317][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:14:51,819][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:14:52,320][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:14:52,821][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:14:53,320][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:14:53,818][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:14:54,320][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:14:54,816][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:14:55,322][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:14:55,817][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:14:56,314][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:14:56,817][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:14:57,313][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:14:57,812][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:14:58,308][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:14:58,805][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:14:59,301][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:14:59,797][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:15:00,296][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:15:00,804][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:15:01,298][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:15:01,794][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:15:02,290][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:15:02,784][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:15:03,279][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:15:03,776][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:15:04,271][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:15:04,768][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:15:05,268][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:15:05,766][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:15:06,271][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:15:06,769][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:15:07,269][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:15:07,767][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:15:08,270][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:15:08,775][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:15:09,275][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:15:09,778][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:15:10,300][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:15:10,806][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:15:11,325][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:15:11,824][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:15:12,325][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:15:12,827][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:15:13,326][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:15:13,823][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:15:14,321][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:15:14,819][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:15:15,328][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:15:15,832][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:15:16,334][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:15:16,834][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10118 tokens.
[2025-11-13 08:15:17,746][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 08:15:18,516][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:15:18,517][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:15:18,519][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:15:19,619][__main__][INFO] - Iteration 628 took 1m 5s (44.19% Gen, 54.13% Train). Generation: 28s, Training: 35s. Estimated remaining time: 44h 28m 31s. Estimated total time: 54h 34m 9s. Time estimates for 10 more iterations: 10m 54s, 100 more iterations: 1h 49m 8s, 500 more iterations: 9h 5m 41s.
[2025-11-13 08:15:19,622][__main__][INFO] - Starting iteration 628.
[2025-11-13 08:15:20,123][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:15:20,123][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:15:35,853][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:15:40,268][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 30 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:15:44,885][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:15:45,722][__main__][INFO] - Number of regex retries in iteration 628: 3
[2025-11-13 08:15:45,723][__main__][INFO] - agents played in iteration 628 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:15:46,679][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:46,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:46,727][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:46,764][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:46,764][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:15:46,765][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:15:47,557][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:15:48,014][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:15:48,517][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:15:49,017][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:15:49,519][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:15:50,019][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:15:50,517][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:15:51,029][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:15:51,525][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:15:52,024][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:15:52,522][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:15:53,022][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:15:53,533][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:15:54,029][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:15:54,524][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:15:55,020][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:15:55,518][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:15:56,025][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:15:56,522][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:15:57,017][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:15:57,512][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:15:58,006][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:15:58,502][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:15:58,997][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:15:59,490][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:16:00,003][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:16:00,498][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:16:00,994][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:16:01,501][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:16:02,001][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:16:02,501][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:16:03,001][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:16:03,496][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:16:03,994][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:16:04,490][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:16:04,985][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:16:05,480][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:16:05,975][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:16:06,475][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:16:06,975][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:16:07,477][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:16:07,978][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:16:08,476][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:16:08,975][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:16:09,473][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:16:09,972][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:16:10,479][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:16:10,982][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:16:11,485][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:16:12,015][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:16:12,517][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:16:13,022][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:16:13,522][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:16:14,024][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:16:14,535][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:16:15,035][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:16:15,538][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:16:16,038][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:16:16,538][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:16:17,042][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:16:17,543][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:16:18,043][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:16:18,549][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:16:19,054][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:16:19,555][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10107 tokens.
[2025-11-13 08:16:20,429][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.02%, ΔTime: 00:00:32
[2025-11-13 08:16:21,206][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:16:21,208][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:16:21,209][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:16:22,389][__main__][INFO] - Iteration 629 took 1m 2s (41.11% Gen, 56.99% Train). Generation: 25s, Training: 35s. Estimated remaining time: 41h 46m 42s. Estimated total time: 51h 53m 23s. Time estimates for 10 more iterations: 10m 22s, 100 more iterations: 1h 43m 46s, 500 more iterations: 8h 38m 53s.
[2025-11-13 08:16:22,392][__main__][INFO] - Starting iteration 629.
[2025-11-13 08:16:22,864][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:16:22,864][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:16:46,744][__main__][INFO] - Number of regex retries in iteration 629: 0
[2025-11-13 08:16:46,746][__main__][INFO] - agents played in iteration 629 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:16:47,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:47,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:47,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:47,741][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:47,742][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:16:47,742][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:16:48,647][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:16:49,104][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:16:49,623][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:16:50,123][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:16:50,626][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:16:51,128][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:16:51,631][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:16:52,139][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:16:52,639][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:16:53,140][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:16:53,641][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:16:54,141][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:16:54,641][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:16:55,143][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:16:55,642][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:16:56,145][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:16:56,642][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:16:57,143][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:16:57,645][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:16:58,141][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:16:58,643][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:16:59,140][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:16:59,638][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:17:00,146][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:17:00,652][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:17:01,151][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:17:01,647][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:17:02,143][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:17:02,643][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:17:03,140][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:17:03,638][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:17:04,145][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:17:04,641][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:17:05,143][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:17:05,641][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:17:06,140][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:17:06,653][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:17:07,154][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:17:07,656][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:17:08,160][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:17:08,661][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:17:09,168][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:17:09,672][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:17:10,175][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:17:10,680][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:17:11,186][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:17:11,688][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:17:12,200][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:17:12,702][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:17:13,205][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:17:13,709][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:17:14,210][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:17:14,711][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:17:15,213][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:17:15,718][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:17:16,218][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:17:16,725][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:17:17,242][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:17:17,742][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:17:18,242][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:17:18,745][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:17:19,244][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:17:19,747][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:17:20,247][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:17:20,754][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10056 tokens.
[2025-11-13 08:17:21,657][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.08%, ΔTime: 00:00:33
[2025-11-13 08:17:22,347][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:17:22,349][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:17:22,350][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:17:23,391][__main__][INFO] - Iteration 630 took 1m 0s (39.46% Gen, 58.82% Train). Generation: 23s, Training: 35s. Estimated remaining time: 40h 18m 41s. Estimated total time: 50h 26m 23s. Time estimates for 10 more iterations: 10m 5s, 100 more iterations: 1h 40m 52s, 500 more iterations: 8h 24m 23s.
[2025-11-13 08:17:23,393][__main__][INFO] - Starting iteration 630.
[2025-11-13 08:17:23,904][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 62 and human policies 1.
[2025-11-13 08:17:23,905][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:17:52,244][__main__][INFO] - Number of regex retries in iteration 630: 0
[2025-11-13 08:17:52,245][__main__][INFO] - agents played in iteration 630 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:17:53,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:53,058][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:53,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:53,107][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:53,108][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:17:53,109][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:17:54,000][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:17:54,458][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:17:54,963][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:17:55,466][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:17:55,968][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:17:56,472][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:17:56,972][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:17:57,474][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:17:57,975][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:17:58,483][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:17:58,986][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:17:59,489][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:17:59,994][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:18:00,497][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:18:01,001][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:18:01,501][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:18:02,004][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:18:02,508][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:18:03,006][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:18:03,504][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:18:04,002][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:18:04,514][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:18:05,012][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:18:05,512][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:18:06,017][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:18:06,517][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:18:07,023][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:18:07,520][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:18:08,021][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:18:08,522][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:18:09,026][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:18:09,525][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:18:10,022][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:18:10,522][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:18:11,038][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:18:11,544][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:18:12,046][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:18:12,547][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:18:13,045][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:18:13,555][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:18:14,058][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:18:14,563][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:18:15,072][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:18:15,579][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:18:16,085][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:18:16,589][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:18:17,094][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:18:17,598][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:18:18,103][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:18:18,607][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:18:19,112][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:18:19,614][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:18:20,117][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:18:20,621][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:18:21,123][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:18:21,641][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:18:22,145][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:18:22,644][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:18:23,144][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:18:23,647][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:18:24,153][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:18:24,656][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:18:25,159][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:18:25,661][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:18:26,162][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10124 tokens.
[2025-11-13 08:18:27,093][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 62.42%, ΔTime: 00:00:33
[2025-11-13 08:18:27,842][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:18:27,844][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:18:27,847][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:18:30,117][__main__][INFO] - Iteration 631 took 1m 6s (42.80% Gen, 53.77% Train). Generation: 28s, Training: 35s. Estimated remaining time: 45h 1m 53s. Estimated total time: 55h 10m 42s. Time estimates for 10 more iterations: 11m 2s, 100 more iterations: 1h 50m 21s, 500 more iterations: 9h 11m 47s.
[2025-11-13 08:18:30,119][__main__][INFO] - Starting iteration 631.
[2025-11-13 08:18:30,609][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:18:30,609][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:18:59,414][__main__][INFO] - Number of regex retries in iteration 631: 0
[2025-11-13 08:18:59,416][__main__][INFO] - agents played in iteration 631 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:19:00,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:00,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:00,299][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:00,321][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:00,322][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:19:00,323][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:19:01,234][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:19:01,691][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:19:02,201][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:19:02,699][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:19:03,199][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:19:03,701][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:19:04,201][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:19:04,702][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:19:05,204][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:19:05,704][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:19:06,213][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:19:06,712][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:19:07,211][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:19:07,710][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:19:08,209][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:19:08,708][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:19:09,208][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:19:09,708][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:19:10,206][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:19:10,707][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:19:11,210][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:19:11,708][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:19:12,207][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:19:12,707][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:19:13,205][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:19:13,701][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:19:14,202][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:19:14,698][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:19:15,196][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:19:15,698][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:19:16,195][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:19:16,693][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:19:17,193][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:19:17,693][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:19:18,197][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:19:18,696][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:19:19,196][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:19:19,702][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:19:20,208][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:19:20,714][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:19:21,219][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:19:21,718][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:19:22,228][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:19:22,729][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:19:23,230][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:19:23,737][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:19:24,240][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:19:24,746][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:19:25,250][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:19:25,757][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:19:26,277][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:19:26,781][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:19:27,287][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:19:27,787][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:19:28,291][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:19:28,796][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:19:29,298][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:19:29,802][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:19:30,309][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:19:30,812][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:19:31,317][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:19:31,819][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:19:32,320][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:19:32,826][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:19:33,327][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10073 tokens.
[2025-11-13 08:19:34,231][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.09%, ΔTime: 00:00:33
[2025-11-13 08:19:34,870][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:19:34,872][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:19:34,874][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:19:35,806][__main__][INFO] - Iteration 632 took 1m 5s (44.18% Gen, 54.39% Train). Generation: 28s, Training: 35s. Estimated remaining time: 44h 10m 0s. Estimated total time: 54h 19m 54s. Time estimates for 10 more iterations: 10m 51s, 100 more iterations: 1h 48m 39s, 500 more iterations: 9h 3m 19s.
[2025-11-13 08:19:35,808][__main__][INFO] - Starting iteration 632.
[2025-11-13 08:19:36,289][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:19:36,289][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:20:06,763][__main__][INFO] - Number of regex retries in iteration 632: 0
[2025-11-13 08:20:06,764][__main__][INFO] - agents played in iteration 632 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:20:07,597][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:07,624][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:07,651][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:07,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:07,675][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:20:07,676][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:20:08,563][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:20:09,018][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:20:09,519][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:20:10,017][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:20:10,521][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:20:11,025][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:20:11,523][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:20:12,023][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:20:12,521][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:20:13,022][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:20:13,521][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:20:14,024][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:20:14,526][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:20:15,031][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:20:15,532][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:20:16,034][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:20:16,536][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:20:17,039][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:20:17,546][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:20:18,045][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:20:18,548][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:20:19,046][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:20:19,545][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:20:20,069][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:20:20,566][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:20:21,065][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:20:21,561][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:20:22,057][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:20:22,556][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:20:23,053][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:20:23,560][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:20:24,057][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:20:24,553][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:20:25,063][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:20:25,565][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:20:26,088][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:20:26,587][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:20:27,091][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:20:27,590][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:20:28,086][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:20:28,596][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:20:29,097][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:20:29,599][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:20:30,101][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:20:30,603][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:20:31,103][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:20:31,607][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:20:32,109][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:20:32,611][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:20:33,112][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:20:33,617][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:20:34,119][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:20:34,620][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:20:35,124][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:20:35,622][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:20:36,122][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:20:36,625][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:20:37,130][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:20:37,632][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:20:38,139][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:20:38,645][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:20:39,147][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:20:39,662][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:20:40,164][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:20:40,678][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10029 tokens.
[2025-11-13 08:20:41,581][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.03%, ΔTime: 00:00:33
[2025-11-13 08:20:42,342][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:20:42,344][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:20:42,345][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:20:43,387][__main__][INFO] - Iteration 633 took 1m 7s (45.42% Gen, 53.03% Train). Generation: 30s, Training: 35s. Estimated remaining time: 45h 43m 54s. Estimated total time: 55h 54m 56s. Time estimates for 10 more iterations: 11m 10s, 100 more iterations: 1h 51m 49s, 500 more iterations: 9h 19m 9s.
[2025-11-13 08:20:43,389][__main__][INFO] - Starting iteration 633.
[2025-11-13 08:20:43,889][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:20:43,890][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:21:05,350][__main__][INFO] - Number of regex retries in iteration 633: 0
[2025-11-13 08:21:05,351][__main__][INFO] - agents played in iteration 633 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:21:06,211][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:06,240][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:06,267][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:06,291][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:06,292][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:21:06,292][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:21:07,179][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:21:07,635][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:21:08,140][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:21:08,639][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:21:09,147][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:21:09,645][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:21:10,147][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:21:10,647][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:21:11,164][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:21:11,667][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:21:12,168][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:21:12,678][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:21:13,179][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:21:13,691][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:21:14,195][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:21:14,705][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:21:15,213][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:21:15,715][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:21:16,220][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:21:16,724][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:21:17,225][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:21:17,728][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:21:18,230][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:21:18,732][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:21:19,235][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:21:19,735][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:21:20,237][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:21:20,739][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:21:21,238][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:21:21,735][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:21:22,232][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:21:22,733][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:21:23,235][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:21:23,740][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:21:24,245][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:21:24,778][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:21:25,280][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:21:25,778][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:21:26,281][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:21:26,782][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:21:27,284][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:21:27,781][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:21:28,289][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:21:28,787][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:21:29,288][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:21:29,789][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:21:30,289][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:21:30,791][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:21:31,292][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:21:31,794][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:21:32,305][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:21:32,804][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:21:33,310][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:21:33,812][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:21:34,317][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:21:34,819][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:21:35,325][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:21:35,829][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:21:36,332][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:21:36,832][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:21:37,336][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:21:37,839][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:21:38,342][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:21:38,847][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:21:39,349][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10085 tokens.
[2025-11-13 08:21:40,286][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.35%, ΔTime: 00:00:33
[2025-11-13 08:21:42,377][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:21:42,379][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:21:42,381][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:21:43,764][__main__][INFO] - Iteration 634 took 59s (35.84% Gen, 61.84% Train). Generation: 21s, Training: 37s. Estimated remaining time: 39h 41m 45s. Estimated total time: 49h 53m 48s. Time estimates for 10 more iterations: 9m 58s, 100 more iterations: 1h 39m 47s, 500 more iterations: 8h 18m 58s.
[2025-11-13 08:21:43,769][__main__][INFO] - Starting iteration 634.
[2025-11-13 08:21:44,251][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:21:44,252][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:22:18,055][__main__][INFO] - Number of regex retries in iteration 634: 0
[2025-11-13 08:22:18,056][__main__][INFO] - agents played in iteration 634 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:22:18,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:18,940][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:18,967][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:18,990][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:18,991][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:22:18,992][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:22:19,868][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:22:20,324][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:22:20,831][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:22:21,331][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:22:21,837][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:22:22,341][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:22:22,842][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:22:23,343][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:22:23,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:22:24,346][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:22:24,854][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:22:25,360][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:22:25,866][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:22:26,370][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:22:26,871][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:22:27,375][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:22:27,876][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:22:28,378][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:22:28,886][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:22:29,390][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:22:29,905][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:22:30,405][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:22:30,905][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:22:31,407][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:22:31,909][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:22:32,418][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:22:32,919][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:22:33,422][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:22:33,930][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:22:34,432][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:22:34,933][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:22:35,436][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:22:35,942][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:22:36,443][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:22:36,944][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:22:37,447][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:22:37,953][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:22:38,452][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:22:38,952][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:22:39,451][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:22:39,958][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:22:40,458][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:22:40,960][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:22:41,460][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:22:41,961][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:22:42,461][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:22:42,963][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:22:43,461][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:22:43,960][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:22:44,462][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:22:44,963][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:22:45,470][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:22:46,001][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:22:46,505][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:22:47,016][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:22:47,521][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:22:48,024][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:22:48,534][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:22:49,037][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:22:49,539][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:22:50,039][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:22:50,539][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:22:51,041][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:22:51,540][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:22:52,041][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10103 tokens.
[2025-11-13 08:22:52,959][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:33
[2025-11-13 08:22:53,739][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:22:53,741][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:22:53,743][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:22:55,035][__main__][INFO] - Iteration 635 took 1m 10s (47.75% Gen, 50.42% Train). Generation: 33s, Training: 35s. Estimated remaining time: 48h 46m 0s. Estimated total time: 58h 59m 13s. Time estimates for 10 more iterations: 11m 47s, 100 more iterations: 1h 57m 58s, 500 more iterations: 9h 49m 52s.
[2025-11-13 08:22:55,037][__main__][INFO] - Starting iteration 635.
[2025-11-13 08:22:55,525][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:22:55,526][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:23:22,801][__main__][INFO] - Number of regex retries in iteration 635: 0
[2025-11-13 08:23:22,802][__main__][INFO] - agents played in iteration 635 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:23:23,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:23,627][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:23,653][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:23,677][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:23,677][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:23:23,678][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:23:24,502][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:23:24,956][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:23:25,465][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:23:25,969][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:23:26,472][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:23:26,978][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:23:27,480][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:23:27,983][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:23:28,493][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:23:28,996][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:23:29,496][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:23:30,000][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:23:30,503][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:23:31,007][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:23:31,513][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:23:32,017][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:23:32,518][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:23:33,019][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:23:33,520][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:23:34,020][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:23:34,528][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:23:35,028][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:23:35,528][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:23:36,038][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:23:36,539][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:23:37,040][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:23:37,543][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:23:38,043][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:23:38,549][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:23:39,053][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:23:39,557][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:23:40,061][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:23:40,562][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:23:41,060][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:23:41,561][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:23:42,062][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:23:42,567][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:23:43,066][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:23:43,567][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:23:44,067][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:23:44,568][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:23:45,070][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:23:45,570][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:23:46,080][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:23:46,585][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:23:47,094][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:23:47,594][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:23:48,117][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:23:48,621][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:23:49,120][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:23:49,621][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:23:50,124][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:23:50,631][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:23:51,130][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:23:51,628][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:23:52,127][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:23:52,624][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:23:53,128][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:23:53,629][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:23:54,129][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:23:54,637][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:23:55,133][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:23:55,631][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:23:56,128][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:23:56,626][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10060 tokens.
[2025-11-13 08:23:57,516][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:33
[2025-11-13 08:23:58,309][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:23:58,310][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:23:58,312][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:23:59,387][__main__][INFO] - Iteration 636 took 1m 3s (42.71% Gen, 55.60% Train). Generation: 27s, Training: 35s. Estimated remaining time: 42h 58m 49s. Estimated total time: 53h 13m 7s. Time estimates for 10 more iterations: 10m 38s, 100 more iterations: 1h 46m 26s, 500 more iterations: 8h 52m 11s.
[2025-11-13 08:23:59,389][__main__][INFO] - Starting iteration 636.
[2025-11-13 08:23:59,872][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:23:59,873][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:24:23,902][__main__][INFO] - Number of regex retries in iteration 636: 0
[2025-11-13 08:24:23,903][__main__][INFO] - agents played in iteration 636 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:24:24,727][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:24,755][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:24,782][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:24,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:24,806][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:24:24,806][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:24:25,581][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:24:26,039][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:24:26,537][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:24:27,044][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:24:27,542][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:24:28,041][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:24:28,548][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:24:29,045][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:24:29,545][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:24:30,047][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:24:30,546][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:24:31,049][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:24:31,550][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:24:32,050][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:24:32,553][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:24:33,057][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:24:33,569][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:24:35,426][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:24:35,974][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:24:36,482][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:24:36,986][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:24:37,490][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:24:37,990][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:24:38,491][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:24:38,993][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:24:39,504][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:24:40,005][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:24:40,529][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:24:41,031][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:24:41,533][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:24:42,038][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:24:42,541][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:24:43,044][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:24:43,545][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:24:44,046][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:24:44,556][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:24:45,057][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:24:45,562][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:24:46,063][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:24:46,567][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:24:47,075][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:24:47,578][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:24:48,082][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:24:48,595][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:24:49,122][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:24:49,625][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:24:50,129][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:24:50,629][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:24:51,143][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:24:51,641][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:24:52,139][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:24:52,639][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:24:53,138][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:24:53,645][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:24:54,144][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:24:54,644][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:24:55,144][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:24:55,645][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:24:56,150][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:24:56,648][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:24:57,151][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:24:57,660][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:24:58,165][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:24:58,664][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:24:59,165][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10160 tokens.
[2025-11-13 08:25:00,083][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:34
[2025-11-13 08:25:00,820][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:25:00,822][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:25:00,824][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:25:01,799][__main__][INFO] - Iteration 637 took 1m 1s (38.80% Gen, 59.62% Train). Generation: 24s, Training: 36s. Estimated remaining time: 41h 21m 2s. Estimated total time: 51h 36m 22s. Time estimates for 10 more iterations: 10m 19s, 100 more iterations: 1h 43m 12s, 500 more iterations: 8h 36m 3s.
[2025-11-13 08:25:01,802][__main__][INFO] - Starting iteration 637.
[2025-11-13 08:25:02,287][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:25:02,288][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:25:32,791][__main__][INFO] - Number of regex retries in iteration 637: 0
[2025-11-13 08:25:32,792][__main__][INFO] - agents played in iteration 637 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:25:33,561][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:33,590][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:33,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:33,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:33,641][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:25:33,642][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:25:34,444][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:25:34,899][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:25:35,407][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:25:35,919][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:25:36,422][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:25:36,948][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:25:37,448][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:25:37,954][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:25:38,464][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:25:38,966][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:25:39,472][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:25:39,973][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:25:40,477][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:25:40,984][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:25:41,486][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:25:41,989][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:25:42,489][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:25:42,993][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:25:43,499][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:25:44,001][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:25:44,505][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:25:45,005][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:25:45,512][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:25:46,013][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:25:46,515][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:25:47,017][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:25:47,528][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:25:48,032][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:25:48,537][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:25:49,041][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:25:49,540][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:25:50,047][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:25:50,547][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:25:51,046][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:25:51,548][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:25:52,051][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:25:52,564][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:25:53,066][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:25:53,567][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:25:54,066][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:25:54,566][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:25:55,069][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:25:55,572][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:25:56,074][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:25:56,576][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:25:57,080][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:25:57,584][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:25:58,088][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:25:58,589][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:25:59,095][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:25:59,595][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:26:00,102][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:26:00,606][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:26:01,105][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:26:01,607][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:26:02,104][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:26:02,604][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:26:03,111][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:26:03,611][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:26:04,111][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:26:04,610][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:26:05,114][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:26:05,641][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:26:06,142][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:26:06,640][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10190 tokens.
[2025-11-13 08:26:07,475][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:33
[2025-11-13 08:26:08,251][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:26:08,253][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:26:08,254][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:26:09,301][__main__][INFO] - Iteration 638 took 1m 7s (45.52% Gen, 52.92% Train). Generation: 30s, Training: 35s. Estimated remaining time: 45h 34m 17s. Estimated total time: 55h 50m 45s. Time estimates for 10 more iterations: 11m 10s, 100 more iterations: 1h 51m 41s, 500 more iterations: 9h 18m 27s.
[2025-11-13 08:26:09,303][__main__][INFO] - Starting iteration 638.
[2025-11-13 08:26:09,789][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:26:09,789][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:26:36,223][__main__][INFO] - Number of regex retries in iteration 638: 0
[2025-11-13 08:26:36,224][__main__][INFO] - agents played in iteration 638 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:26:37,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:37,183][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:37,210][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:37,233][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:37,233][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:26:37,234][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:26:38,019][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:26:38,479][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:26:38,988][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:26:39,492][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:26:39,994][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:26:40,496][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:26:41,003][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:26:41,503][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:26:42,001][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:26:42,502][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:26:43,004][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:26:43,504][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:26:44,004][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:26:44,504][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:26:45,005][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:26:45,507][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:26:46,010][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:26:46,511][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:26:47,011][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:26:47,513][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:26:48,013][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:26:48,515][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:26:49,020][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:26:49,523][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:26:50,022][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:26:50,529][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:26:51,040][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:26:51,560][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:26:52,063][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:26:52,569][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:26:53,069][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:26:53,571][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:26:54,079][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:26:54,583][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:26:55,088][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:26:55,592][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:26:56,096][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:26:56,599][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:26:57,099][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:26:57,601][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:26:58,105][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:26:58,606][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:26:59,107][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:26:59,609][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:27:00,110][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:27:00,609][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:27:02,112][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:27:02,623][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:27:03,125][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:27:03,659][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:27:04,161][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:27:04,662][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:27:05,164][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:27:05,667][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:27:06,190][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:27:06,690][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:27:07,191][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:27:07,692][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:27:08,188][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:27:08,690][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:27:09,186][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:27:09,684][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:27:10,188][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:27:10,686][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:27:11,184][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10125 tokens.
[2025-11-13 08:27:12,084][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:34
[2025-11-13 08:27:12,756][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:27:12,758][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:27:12,759][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:27:13,747][__main__][INFO] - Iteration 639 took 1m 3s (41.33% Gen, 57.12% Train). Generation: 26s, Training: 36s. Estimated remaining time: 43h 0m 23s. Estimated total time: 53h 17m 56s. Time estimates for 10 more iterations: 10m 39s, 100 more iterations: 1h 46m 35s, 500 more iterations: 8h 52m 59s.
[2025-11-13 08:27:13,749][__main__][INFO] - Starting iteration 639.
[2025-11-13 08:27:14,234][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:27:14,234][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:27:50,935][__main__][INFO] - Number of regex retries in iteration 639: 0
[2025-11-13 08:27:50,936][__main__][INFO] - agents played in iteration 639 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:27:51,778][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:51,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:51,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:51,858][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:51,858][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:27:51,859][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:27:52,745][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:27:53,212][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:27:53,718][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:27:54,222][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:27:54,724][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:27:55,231][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:27:55,736][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:27:56,240][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:27:56,743][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:27:57,245][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:27:57,745][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:27:58,245][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:27:58,758][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:27:59,260][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:27:59,787][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:28:00,290][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:28:00,790][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:28:01,295][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:28:01,796][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:28:02,304][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:28:02,805][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:28:03,306][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:28:03,809][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:28:04,309][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:28:04,815][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:28:05,314][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:28:05,814][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:28:06,316][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:28:06,817][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:28:07,316][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:28:07,817][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:28:08,320][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:28:08,821][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:28:09,322][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:28:09,820][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:28:10,321][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:28:10,823][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:28:11,321][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:28:11,822][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:28:12,325][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:28:12,822][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:28:13,319][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:28:13,818][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:28:14,322][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:28:14,822][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:28:15,322][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:28:15,816][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:28:16,315][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:28:16,816][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:28:17,317][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:28:17,819][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:28:18,323][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:28:18,823][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:28:19,321][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:28:19,821][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:28:20,322][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:28:20,828][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:28:21,326][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:28:21,821][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:28:22,322][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:28:22,816][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:28:23,316][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:28:23,814][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:28:24,309][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:28:24,810][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10085 tokens.
[2025-11-13 08:28:25,623][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.12%, ΔTime: 00:00:32
[2025-11-13 08:28:26,429][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:28:26,430][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:28:26,432][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:28:27,471][__main__][INFO] - Iteration 640 took 1m 13s (50.11% Gen, 48.47% Train). Generation: 36s, Training: 35s. Estimated remaining time: 50h 43m 8s. Estimated total time: 61h 1m 54s. Time estimates for 10 more iterations: 12m 12s, 100 more iterations: 2h 2m 3s, 500 more iterations: 10h 10m 19s.
[2025-11-13 08:28:27,473][__main__][INFO] - Starting iteration 640.
[2025-11-13 08:28:27,960][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 63 and human policies 1.
[2025-11-13 08:28:27,961][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:28:49,998][__main__][INFO] - Number of regex retries in iteration 640: 0
[2025-11-13 08:28:49,999][__main__][INFO] - agents played in iteration 640 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:28:50,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:50,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:51,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:51,024][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:51,024][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:28:51,025][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:28:51,892][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:28:52,357][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:28:52,863][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:28:53,374][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:28:53,874][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:28:54,376][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:28:54,894][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:28:55,393][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:28:55,894][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:28:56,400][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:28:56,907][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:28:57,415][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:28:57,918][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:28:58,421][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:28:58,923][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:28:59,426][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:28:59,936][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:29:00,439][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:29:00,941][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:29:01,444][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:29:01,946][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:29:02,451][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:29:02,954][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:29:03,457][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:29:03,960][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:29:04,461][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:29:04,964][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:29:05,500][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:29:06,006][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:29:06,512][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:29:07,016][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:29:07,517][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:29:08,020][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:29:08,520][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:29:09,024][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:29:09,525][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:29:10,023][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:29:10,524][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:29:11,022][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:29:11,522][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:29:12,021][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:29:12,521][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:29:13,021][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:29:13,522][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:29:14,022][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:29:14,526][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:29:15,026][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:29:15,525][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:29:16,024][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:29:16,525][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:29:17,024][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:29:17,521][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:29:18,019][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:29:18,521][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:29:19,019][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:29:19,516][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:29:20,016][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:29:20,513][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:29:21,018][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:29:21,516][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:29:22,013][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:29:22,514][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:29:23,015][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:29:23,516][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:29:24,018][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10061 tokens.
[2025-11-13 08:29:24,824][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.11%, ΔTime: 00:00:32
[2025-11-13 08:29:25,602][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:29:25,603][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:29:25,605][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:29:27,647][__main__][INFO] - Iteration 641 took 59s (36.92% Gen, 59.65% Train). Generation: 22s, Training: 35s. Estimated remaining time: 39h 24m 37s. Estimated total time: 49h 44m 23s. Time estimates for 10 more iterations: 9m 56s, 100 more iterations: 1h 39m 28s, 500 more iterations: 8h 17m 23s.
[2025-11-13 08:29:27,650][__main__][INFO] - Starting iteration 641.
[2025-11-13 08:29:28,140][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:29:28,141][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:29:59,031][__main__][INFO] - Number of regex retries in iteration 641: 0
[2025-11-13 08:29:59,032][__main__][INFO] - agents played in iteration 641 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:29:59,909][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:59,937][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:59,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:59,988][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:59,989][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:29:59,989][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:30:00,936][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:30:01,397][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:30:01,914][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:30:02,417][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:30:02,926][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:30:03,430][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:30:03,932][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:30:04,433][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:30:04,933][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:30:05,439][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:30:05,949][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:30:06,453][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:30:06,962][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:30:07,465][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:30:07,967][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:30:08,473][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:30:08,983][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:30:09,485][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:30:09,995][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:30:10,500][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:30:11,026][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:30:11,526][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:30:12,027][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:30:12,528][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:30:13,030][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:30:13,536][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:30:14,041][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:30:14,545][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:30:15,047][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:30:15,550][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:30:16,051][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:30:16,560][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:30:17,059][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:30:17,565][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:30:18,068][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:30:18,569][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:30:19,076][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:30:19,576][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:30:20,081][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:30:20,582][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:30:21,083][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:30:21,585][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:30:22,084][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:30:22,584][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:30:23,084][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:30:23,584][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:30:24,085][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:30:24,582][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:30:25,082][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:30:25,587][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:30:26,085][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:30:26,589][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:30:27,090][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:30:27,590][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:30:28,091][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:30:28,588][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:30:29,091][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:30:29,588][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:30:30,085][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:30:30,587][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:30:31,083][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:30:31,580][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:30:32,080][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:30:32,577][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:30:33,078][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10153 tokens.
[2025-11-13 08:30:33,900][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:32
[2025-11-13 08:30:34,690][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:30:34,691][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:30:34,693][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:30:35,724][__main__][INFO] - Iteration 642 took 1m 7s (45.71% Gen, 52.76% Train). Generation: 30s, Training: 35s. Estimated remaining time: 45h 58m 19s. Estimated total time: 56h 19m 13s. Time estimates for 10 more iterations: 11m 15s, 100 more iterations: 1h 52m 38s, 500 more iterations: 9h 23m 12s.
[2025-11-13 08:30:35,726][__main__][INFO] - Starting iteration 642.
[2025-11-13 08:30:36,215][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:30:36,216][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:31:02,529][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:31:06,710][__main__][INFO] - Number of regex retries in iteration 642: 1
[2025-11-13 08:31:06,710][__main__][INFO] - agents played in iteration 642 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:31:07,633][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:07,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:07,680][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:07,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:07,705][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:31:07,705][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:31:08,580][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:31:09,037][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:31:09,545][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:31:10,050][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:31:10,560][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:31:11,062][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:31:11,564][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:31:12,065][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:31:12,568][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:31:13,073][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:31:13,573][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:31:14,076][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:31:14,577][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:31:15,085][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:31:15,590][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:31:16,098][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:31:16,604][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:31:17,107][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:31:17,607][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:31:18,107][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:31:18,611][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:31:19,116][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:31:19,619][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:31:20,126][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:31:20,631][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:31:21,130][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:31:21,647][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:31:22,147][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:31:22,653][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:31:23,154][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:31:23,656][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:31:24,190][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:31:24,691][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:31:25,193][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:31:25,698][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:31:26,200][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:31:26,705][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:31:27,208][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:31:27,709][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:31:28,211][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:31:28,711][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:31:29,214][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:31:29,731][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:31:30,236][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:31:30,746][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:31:31,242][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:31:31,740][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:31:32,255][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:31:32,755][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:31:33,251][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:31:33,746][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:31:34,241][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:31:34,747][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:31:35,243][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:31:35,743][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:31:36,242][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:31:36,741][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:31:37,238][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:31:37,737][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:31:38,236][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:31:38,748][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:31:39,249][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:31:39,750][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:31:40,251][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:31:40,751][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10144 tokens.
[2025-11-13 08:31:41,567][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-13 08:31:42,342][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:31:42,343][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:31:42,345][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:31:43,424][__main__][INFO] - Iteration 643 took 1m 7s (45.37% Gen, 53.02% Train). Generation: 30s, Training: 35s. Estimated remaining time: 45h 38m 24s. Estimated total time: 56h 0m 26s. Time estimates for 10 more iterations: 11m 12s, 100 more iterations: 1h 52m 0s, 500 more iterations: 9h 20m 4s.
[2025-11-13 08:31:43,426][__main__][INFO] - Starting iteration 643.
[2025-11-13 08:31:43,918][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:31:43,919][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:32:07,334][__main__][INFO] - Number of regex retries in iteration 643: 0
[2025-11-13 08:32:07,334][__main__][INFO] - agents played in iteration 643 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:32:08,188][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,210][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,233][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,256][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,256][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:32:08,258][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:32:09,138][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:32:09,596][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:32:10,101][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:32:10,604][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:32:11,105][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:32:11,615][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:32:12,114][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:32:12,629][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:32:14,174][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:32:14,673][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:32:15,177][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:32:15,686][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:32:16,186][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:32:16,690][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:32:17,190][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:32:17,697][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:32:18,200][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:32:18,700][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:32:19,201][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:32:19,704][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:32:20,210][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:32:20,715][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:32:21,228][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:32:21,728][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:32:22,253][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:32:22,759][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:32:23,266][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:32:23,768][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:32:24,273][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:32:24,779][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:32:25,303][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:32:25,811][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:32:26,316][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:32:26,818][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:32:27,316][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:32:27,826][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:32:28,324][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:32:28,831][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:32:29,332][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:32:29,831][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:32:30,350][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:32:30,850][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:32:31,351][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:32:31,858][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:32:32,362][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:32:32,873][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:32:33,375][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:32:33,879][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:32:34,381][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:32:34,880][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:32:35,381][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:32:35,883][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:32:36,387][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:32:36,887][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:32:37,387][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:32:37,890][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:32:38,395][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:32:38,897][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:32:39,400][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:32:39,903][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:32:40,403][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:32:40,907][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:32:41,407][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:32:41,906][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:32:42,408][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10160 tokens.
[2025-11-13 08:32:43,263][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:34
[2025-11-13 08:32:43,981][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:32:43,982][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:32:43,985][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:32:45,151][__main__][INFO] - Iteration 644 took 1m 1s (38.24% Gen, 59.85% Train). Generation: 23s, Training: 36s. Estimated remaining time: 40h 38m 36s. Estimated total time: 51h 1m 39s. Time estimates for 10 more iterations: 10m 12s, 100 more iterations: 1h 42m 3s, 500 more iterations: 8h 30m 16s.
[2025-11-13 08:32:45,153][__main__][INFO] - Starting iteration 644.
[2025-11-13 08:32:45,629][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:32:45,629][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:33:15,174][__main__][INFO] - Number of regex retries in iteration 644: 0
[2025-11-13 08:33:15,175][__main__][INFO] - agents played in iteration 644 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:33:16,088][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:16,115][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:16,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:16,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:16,164][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:33:16,165][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:33:17,038][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:33:17,499][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:33:18,004][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:33:18,509][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:33:19,008][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:33:19,510][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:33:20,011][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:33:20,512][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:33:21,012][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:33:21,520][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:33:22,021][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:33:22,523][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:33:23,024][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:33:23,527][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:33:24,041][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:33:24,543][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:33:25,044][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:33:25,547][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:33:26,049][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:33:26,569][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:33:27,068][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:33:27,569][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:33:28,075][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:33:28,576][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:33:29,085][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:33:29,596][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:33:30,100][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:33:30,605][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:33:31,109][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:33:31,615][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:33:32,145][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:33:32,645][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:33:33,148][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:33:33,648][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:33:34,147][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:33:34,661][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:33:35,163][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:33:35,669][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:33:36,168][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:33:36,670][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:33:37,173][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:33:37,677][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:33:38,176][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:33:38,681][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:33:39,181][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:33:39,682][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:33:40,183][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:33:40,685][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:33:41,193][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:33:41,696][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:33:42,198][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:33:42,700][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:33:43,201][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:33:43,704][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:33:44,204][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:33:44,704][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:33:45,206][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:33:45,713][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:33:46,212][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:33:46,718][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:33:47,222][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:33:47,726][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:33:48,225][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:33:48,720][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:33:49,224][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10043 tokens.
[2025-11-13 08:33:50,057][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:33
[2025-11-13 08:33:50,841][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:33:50,844][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:33:50,845][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:33:51,851][__main__][INFO] - Iteration 645 took 1m 6s (44.61% Gen, 53.86% Train). Generation: 29s, Training: 35s. Estimated remaining time: 44h 46m 57s. Estimated total time: 55h 11m 7s. Time estimates for 10 more iterations: 11m 2s, 100 more iterations: 1h 50m 22s, 500 more iterations: 9h 11m 51s.
[2025-11-13 08:33:51,853][__main__][INFO] - Starting iteration 645.
[2025-11-13 08:33:52,348][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:33:52,348][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:34:18,920][__main__][INFO] - Number of regex retries in iteration 645: 0
[2025-11-13 08:34:18,921][__main__][INFO] - agents played in iteration 645 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:34:19,773][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:19,800][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:19,827][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:19,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:19,851][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:34:19,852][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:34:20,695][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:34:21,150][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:34:21,656][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:34:22,158][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:34:22,660][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:34:23,178][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:34:23,680][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:34:24,180][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:34:24,682][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:34:25,183][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:34:25,689][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:34:26,199][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:34:26,701][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:34:27,204][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:34:27,703][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:34:28,208][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:34:28,705][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:34:29,205][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:34:29,710][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:34:30,208][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:34:30,715][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:34:31,214][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:34:31,713][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:34:32,216][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:34:32,719][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:34:33,224][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:34:33,728][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:34:34,232][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:34:34,738][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:34:35,241][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:34:35,744][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:34:36,250][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:34:36,756][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:34:37,255][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:34:37,768][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:34:38,269][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:34:38,781][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:34:39,282][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:34:39,784][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:34:40,289][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:34:40,790][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:34:41,290][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:34:41,792][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:34:42,291][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:34:42,800][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:34:43,301][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:34:43,803][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:34:44,303][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:34:44,804][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:34:45,338][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:34:45,838][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:34:46,345][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:34:46,850][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:34:47,352][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:34:47,853][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:34:48,368][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:34:48,868][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:34:49,368][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:34:49,865][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:34:50,363][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:34:50,879][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:34:51,377][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:34:51,876][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:34:52,377][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:34:52,875][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10049 tokens.
[2025-11-13 08:34:53,715][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:33
[2025-11-13 08:34:54,469][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:34:54,471][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:34:54,474][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:34:55,677][__main__][INFO] - Iteration 646 took 1m 3s (41.96% Gen, 56.14% Train). Generation: 26s, Training: 35s. Estimated remaining time: 42h 21m 15s. Estimated total time: 52h 46m 30s. Time estimates for 10 more iterations: 10m 33s, 100 more iterations: 1h 45m 33s, 500 more iterations: 8h 47m 45s.
[2025-11-13 08:34:55,679][__main__][INFO] - Starting iteration 646.
[2025-11-13 08:34:56,160][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:34:56,161][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:35:28,281][__main__][INFO] - Number of regex retries in iteration 646: 0
[2025-11-13 08:35:28,282][__main__][INFO] - agents played in iteration 646 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:35:29,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:29,236][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:29,260][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:29,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:29,284][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:35:29,284][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:35:30,186][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:35:30,645][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:35:31,151][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:35:31,658][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:35:32,162][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:35:32,677][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:35:33,178][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:35:33,682][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:35:34,184][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:35:34,686][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:35:35,189][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:35:35,692][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:35:36,194][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:35:36,705][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:35:37,206][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:35:37,707][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:35:38,213][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:35:38,718][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:35:39,220][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:35:39,721][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:35:40,221][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:35:40,723][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:35:41,222][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:35:41,723][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:35:42,225][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:35:42,728][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:35:43,231][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:35:43,731][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:35:44,231][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:35:44,734][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:35:45,234][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:35:45,734][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:35:46,235][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:35:46,740][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:35:47,243][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:35:47,746][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:35:48,247][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:35:48,756][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:35:49,260][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:35:49,787][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:35:50,290][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:35:50,792][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:35:51,293][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:35:51,791][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:35:52,297][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:35:52,796][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:35:53,292][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:35:53,790][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:35:54,288][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:35:54,786][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:35:55,284][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:35:55,783][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:35:56,300][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:35:56,799][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:35:57,298][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:35:57,801][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:35:58,303][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:35:58,802][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:35:59,303][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:35:59,805][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:36:00,314][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:36:00,811][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:36:01,310][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:36:01,809][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:36:02,306][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9993 tokens.
[2025-11-13 08:36:03,141][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 61.98%, ΔTime: 00:00:32
[2025-11-13 08:36:03,923][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:36:03,925][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:36:03,927][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:36:04,931][__main__][INFO] - Iteration 647 took 1m 8s (46.71% Gen, 51.83% Train). Generation: 32s, Training: 35s. Estimated remaining time: 46h 52m 12s. Estimated total time: 57h 18m 35s. Time estimates for 10 more iterations: 11m 27s, 100 more iterations: 1h 54m 37s, 500 more iterations: 9h 33m 5s.
[2025-11-13 08:36:04,933][__main__][INFO] - Starting iteration 647.
[2025-11-13 08:36:05,416][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:36:05,417][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:36:35,906][__main__][INFO] - Number of regex retries in iteration 647: 0
[2025-11-13 08:36:35,907][__main__][INFO] - agents played in iteration 647 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:36:36,744][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:36,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:36,792][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:36,815][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:36,816][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:36:36,817][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:36:37,707][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:36:38,165][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:36:38,679][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:36:39,180][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:36:39,685][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:36:40,202][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:36:40,706][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:36:41,207][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:36:41,710][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:36:42,213][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:36:42,729][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:36:43,231][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:36:43,736][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:36:44,247][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:36:44,752][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:36:45,259][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:36:45,763][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:36:46,265][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:36:46,772][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:36:47,273][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:36:47,774][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:36:48,276][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:36:48,778][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:36:49,283][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:36:49,790][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:36:50,295][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:36:50,806][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:36:51,314][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:36:51,816][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:36:52,319][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:36:52,821][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:36:53,340][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:36:53,845][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:36:54,348][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:36:54,851][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:36:55,352][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:36:55,853][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:36:56,353][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:36:56,851][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:36:57,355][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:36:57,855][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:36:58,356][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:36:58,860][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:36:59,365][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:36:59,870][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:37:00,370][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:37:00,869][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:37:01,369][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:37:01,869][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:37:02,369][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:37:02,871][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:37:03,375][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:37:03,878][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:37:04,382][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:37:04,881][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:37:05,383][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:37:05,885][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:37:06,402][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:37:06,905][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:37:07,406][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:37:07,923][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:37:08,426][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:37:08,927][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:37:09,427][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:37:09,925][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10127 tokens.
[2025-11-13 08:37:10,804][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:33
[2025-11-13 08:37:11,583][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:37:11,585][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:37:11,586][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:37:12,685][__main__][INFO] - Iteration 648 took 1m 7s (45.32% Gen, 53.04% Train). Generation: 30s, Training: 35s. Estimated remaining time: 45h 35m 56s. Estimated total time: 56h 3m 27s. Time estimates for 10 more iterations: 11m 12s, 100 more iterations: 1h 52m 6s, 500 more iterations: 9h 20m 34s.
[2025-11-13 08:37:12,687][__main__][INFO] - Starting iteration 648.
[2025-11-13 08:37:13,176][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:37:13,177][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:37:34,914][__main__][INFO] - Number of regex retries in iteration 648: 0
[2025-11-13 08:37:34,914][__main__][INFO] - agents played in iteration 648 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:37:35,744][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:35,771][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:35,798][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:35,821][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:35,822][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:37:35,822][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:37:36,618][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:37:37,072][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:37:37,574][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:37:38,073][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:37:38,569][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:37:39,066][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:37:39,570][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:37:40,068][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:37:40,572][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:37:41,083][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:37:41,588][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:37:42,095][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:37:42,602][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:37:43,103][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:37:43,607][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:37:44,108][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:37:44,634][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:37:45,138][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:37:46,368][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:37:46,913][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:37:47,423][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:37:47,926][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:37:48,444][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:37:48,948][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:37:49,453][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:37:49,955][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:37:50,464][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:37:50,973][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:37:51,479][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:37:51,986][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:37:52,501][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:37:53,004][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:37:53,511][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:37:54,031][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:37:54,539][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:37:55,055][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:37:55,572][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:37:56,080][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:37:56,586][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:37:57,089][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:37:57,594][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:37:58,100][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:37:58,601][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:37:59,104][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:37:59,606][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:38:00,108][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:38:00,616][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:38:01,120][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:38:01,637][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:38:02,142][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:38:02,645][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:38:03,148][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:38:03,650][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:38:04,155][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:38:04,656][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:38:05,158][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:38:05,677][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:38:06,181][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:38:06,685][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:38:07,192][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:38:07,695][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:38:08,200][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:38:08,703][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:38:09,210][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:38:09,714][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10056 tokens.
[2025-11-13 08:38:10,611][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:34
[2025-11-13 08:38:11,327][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:38:11,329][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:38:11,331][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:38:12,343][__main__][INFO] - Iteration 649 took 59s (36.74% Gen, 61.55% Train). Generation: 21s, Training: 36s. Estimated remaining time: 38h 49m 52s. Estimated total time: 49h 18m 23s. Time estimates for 10 more iterations: 9m 51s, 100 more iterations: 1h 38m 36s, 500 more iterations: 8h 13m 3s.
[2025-11-13 08:38:12,346][__main__][INFO] - Starting iteration 649.
[2025-11-13 08:38:12,839][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:38:12,840][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:38:45,830][__main__][INFO] - Number of regex retries in iteration 649: 0
[2025-11-13 08:38:45,831][__main__][INFO] - agents played in iteration 649 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:38:46,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:46,691][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:46,719][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:46,743][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:46,744][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:38:46,744][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:38:47,644][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:38:48,102][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:38:48,608][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:38:49,124][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:38:49,627][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:38:50,148][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:38:50,648][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:38:51,150][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:38:51,663][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:38:52,164][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:38:52,666][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:38:53,166][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:38:53,668][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:38:54,174][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:38:54,676][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:38:55,181][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:38:55,687][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:38:56,187][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:38:56,691][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:38:57,194][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:38:57,696][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:38:58,198][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:38:58,701][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:38:59,204][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:38:59,716][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:39:00,218][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:39:00,721][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:39:01,225][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:39:01,728][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:39:02,250][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:39:02,751][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:39:03,252][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:39:03,755][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:39:04,261][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:39:04,768][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:39:05,270][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:39:05,770][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:39:06,280][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:39:06,782][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:39:07,287][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:39:07,789][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:39:08,291][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:39:08,793][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:39:09,293][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:39:09,799][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:39:10,296][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:39:10,793][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:39:11,290][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:39:11,792][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:39:12,292][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:39:12,797][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:39:13,298][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:39:13,797][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:39:14,302][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:39:14,802][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:39:15,304][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:39:15,802][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:39:16,305][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:39:16,803][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:39:17,299][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:39:17,800][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:39:18,297][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:39:18,798][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:39:19,295][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:39:19,796][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10126 tokens.
[2025-11-13 08:39:20,675][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:33
[2025-11-13 08:39:21,424][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:39:21,426][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:39:21,427][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:39:22,449][__main__][INFO] - Iteration 650 took 1m 9s (47.39% Gen, 51.14% Train). Generation: 32s, Training: 35s. Estimated remaining time: 47h 30m 50s. Estimated total time: 58h 0m 30s. Time estimates for 10 more iterations: 11m 36s, 100 more iterations: 1h 56m 1s, 500 more iterations: 9h 40m 5s.
[2025-11-13 08:39:22,451][__main__][INFO] - Starting iteration 650.
[2025-11-13 08:39:22,936][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 64 and human policies 1.
[2025-11-13 08:39:22,936][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:39:53,972][__main__][INFO] - Number of regex retries in iteration 650: 0
[2025-11-13 08:39:53,972][__main__][INFO] - agents played in iteration 650 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:39:54,831][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:54,854][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:54,879][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:54,901][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:54,902][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:39:54,902][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:39:55,778][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:39:56,245][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:39:56,752][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:39:57,258][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:39:57,764][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:39:58,274][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:39:58,783][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:39:59,291][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:39:59,792][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:40:00,294][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:40:00,795][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:40:01,296][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:40:02,784][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:40:03,304][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:40:03,816][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:40:04,331][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:40:04,831][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:40:05,353][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:40:05,857][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:40:06,359][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:40:06,870][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:40:07,372][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:40:07,875][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:40:08,380][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:40:08,884][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:40:09,387][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:40:09,887][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:40:10,389][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:40:10,891][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:40:11,394][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:40:11,896][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:40:12,398][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:40:12,900][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:40:13,407][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:40:13,910][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:40:14,411][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:40:14,915][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:40:15,416][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:40:15,919][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:40:16,420][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:40:16,923][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:40:17,420][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:40:17,918][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:40:18,415][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:40:18,913][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:40:19,412][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:40:19,923][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:40:20,423][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:40:20,922][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:40:21,423][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:40:21,923][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:40:22,439][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:40:22,937][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:40:23,439][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:40:23,938][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:40:24,436][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:40:24,947][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:40:25,443][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:40:25,940][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:40:26,440][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:40:26,939][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:40:27,436][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:40:27,932][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:40:28,428][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:40:28,941][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10098 tokens.
[2025-11-13 08:40:29,861][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:34
[2025-11-13 08:40:30,522][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:40:30,524][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:40:30,525][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:40:32,645][__main__][INFO] - Iteration 651 took 1m 9s (44.52% Gen, 52.43% Train). Generation: 31s, Training: 36s. Estimated remaining time: 47h 34m 40s. Estimated total time: 58h 5m 31s. Time estimates for 10 more iterations: 11m 37s, 100 more iterations: 1h 56m 11s, 500 more iterations: 9h 40m 55s.
[2025-11-13 08:40:32,648][__main__][INFO] - Starting iteration 651.
[2025-11-13 08:40:33,134][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:40:33,134][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:41:02,122][__main__][INFO] - Number of regex retries in iteration 651: 0
[2025-11-13 08:41:02,123][__main__][INFO] - agents played in iteration 651 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:41:02,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:03,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:03,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:03,058][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:03,059][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:41:03,060][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:41:03,993][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:41:04,456][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:41:04,965][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:41:05,474][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:41:05,977][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:41:06,479][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:41:06,990][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:41:07,494][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:41:07,997][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:41:08,502][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:41:09,005][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:41:09,510][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:41:10,017][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:41:10,520][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:41:11,030][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:41:11,537][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:41:12,053][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:41:12,553][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:41:13,057][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:41:13,558][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:41:14,060][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:41:14,570][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:41:15,070][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:41:15,577][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:41:16,083][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:41:16,587][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:41:17,089][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:41:17,592][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:41:18,092][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:41:18,593][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:41:19,099][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:41:19,604][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:41:20,104][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:41:20,609][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:41:21,143][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:41:21,648][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:41:22,153][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:41:22,673][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:41:23,179][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:41:23,683][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:41:24,187][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:41:24,691][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:41:25,199][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:41:25,703][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:41:26,202][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:41:26,703][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:41:27,206][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:41:27,709][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:41:28,210][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:41:28,711][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:41:29,209][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:41:29,706][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:41:30,205][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:41:30,706][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:41:31,203][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:41:31,717][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:41:32,224][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:41:32,730][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:41:33,237][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:41:33,743][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:41:34,243][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:41:34,743][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:41:35,241][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:41:35,745][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:41:36,242][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10166 tokens.
[2025-11-13 08:41:37,126][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:33
[2025-11-13 08:41:37,886][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:41:37,888][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:41:37,889][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:41:38,903][__main__][INFO] - Iteration 652 took 1m 5s (44.08% Gen, 54.38% Train). Generation: 28s, Training: 35s. Estimated remaining time: 44h 16m 30s. Estimated total time: 54h 48m 28s. Time estimates for 10 more iterations: 10m 57s, 100 more iterations: 1h 49m 36s, 500 more iterations: 9h 8m 4s.
[2025-11-13 08:41:38,905][__main__][INFO] - Starting iteration 652.
[2025-11-13 08:41:39,378][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:41:39,379][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:42:03,732][__main__][INFO] - Number of regex retries in iteration 652: 0
[2025-11-13 08:42:03,733][__main__][INFO] - agents played in iteration 652 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:42:04,583][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:04,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:04,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:04,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:04,665][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:42:04,666][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:42:05,556][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:42:06,012][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:42:06,522][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:42:07,031][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:42:07,531][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:42:08,032][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:42:08,534][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:42:09,033][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:42:09,532][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:42:10,035][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:42:10,539][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:42:11,049][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:42:11,548][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:42:12,062][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:42:12,564][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:42:13,065][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:42:13,570][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:42:14,074][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:42:14,574][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:42:15,079][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:42:15,581][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:42:16,087][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:42:16,588][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:42:17,091][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:42:17,600][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:42:18,105][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:42:18,605][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:42:19,109][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:42:19,609][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:42:20,111][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:42:20,613][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:42:21,115][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:42:21,616][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:42:22,120][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:42:22,624][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:42:23,134][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:42:23,640][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:42:24,154][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:42:24,660][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:42:25,165][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:42:25,672][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:42:26,171][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:42:26,679][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:42:27,179][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:42:27,681][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:42:28,181][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:42:28,681][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:42:29,180][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:42:29,679][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:42:30,179][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:42:30,689][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:42:31,188][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:42:31,686][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:42:32,186][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:42:32,684][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:42:33,185][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:42:33,687][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:42:34,184][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:42:34,690][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:42:35,188][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:42:35,689][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:42:36,186][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:42:36,686][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:42:37,190][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:42:37,690][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10103 tokens.
[2025-11-13 08:42:38,545][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:33
[2025-11-13 08:42:39,318][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:42:39,320][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:42:39,321][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:42:40,320][__main__][INFO] - Iteration 653 took 1m 0s (39.96% Gen, 58.40% Train). Generation: 24s, Training: 35s. Estimated remaining time: 40h 14m 9s. Estimated total time: 50h 47m 7s. Time estimates for 10 more iterations: 10m 9s, 100 more iterations: 1h 41m 34s, 500 more iterations: 8h 27m 51s.
[2025-11-13 08:42:40,322][__main__][INFO] - Starting iteration 653.
[2025-11-13 08:42:40,822][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:42:40,823][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:43:10,820][__main__][INFO] - Number of regex retries in iteration 653: 0
[2025-11-13 08:43:10,820][__main__][INFO] - agents played in iteration 653 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:43:11,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:11,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:11,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:11,761][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:11,762][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:43:11,762][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:43:12,651][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:43:13,106][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:43:13,616][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:43:14,118][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:43:14,621][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:43:15,128][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:43:15,632][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:43:16,134][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:43:16,642][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:43:17,144][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:43:17,648][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:43:18,151][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:43:18,658][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:43:19,163][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:43:19,666][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:43:20,172][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:43:20,674][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:43:21,175][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:43:21,693][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:43:22,191][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:43:22,694][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:43:23,196][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:43:23,695][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:43:24,199][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:43:24,696][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:43:25,200][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:43:25,703][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:43:26,203][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:43:26,705][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:43:27,207][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:43:27,705][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:43:28,210][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:43:28,708][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:43:29,210][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:43:29,716][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:43:30,214][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:43:30,718][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:43:31,223][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:43:31,727][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:43:32,233][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:43:32,737][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:43:33,242][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:43:33,745][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:43:34,253][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:43:34,757][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:43:35,260][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:43:35,760][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:43:36,274][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:43:36,780][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:43:37,283][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:43:37,783][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:43:38,279][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:43:38,780][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:43:39,280][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:43:39,783][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:43:40,280][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:43:40,780][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:43:41,297][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:43:41,798][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:43:42,298][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:43:42,801][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:43:43,301][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:43:43,801][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:43:44,301][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:43:44,802][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10144 tokens.
[2025-11-13 08:43:45,655][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:33
[2025-11-13 08:43:46,419][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:43:46,420][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:43:46,422][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:43:47,603][__main__][INFO] - Iteration 654 took 1m 6s (44.92% Gen, 53.31% Train). Generation: 29s, Training: 35s. Estimated remaining time: 45h 4m 58s. Estimated total time: 55h 39m 4s. Time estimates for 10 more iterations: 11m 7s, 100 more iterations: 1h 51m 18s, 500 more iterations: 9h 16m 30s.
[2025-11-13 08:43:47,605][__main__][INFO] - Starting iteration 654.
[2025-11-13 08:43:48,093][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:43:48,093][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:44:17,990][__main__][INFO] - Number of regex retries in iteration 654: 0
[2025-11-13 08:44:17,991][__main__][INFO] - agents played in iteration 654 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:44:18,846][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:18,871][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:18,897][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:18,921][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:18,922][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:44:18,923][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:44:19,797][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:44:20,255][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:44:20,770][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:44:21,276][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:44:21,783][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:44:22,286][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:44:22,788][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:44:23,311][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:44:23,815][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:44:24,316][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:44:24,818][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:44:25,326][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:44:25,831][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:44:26,332][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:44:26,836][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:44:27,348][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:44:27,850][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:44:28,355][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:44:28,860][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:44:29,364][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:44:29,881][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:44:30,385][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:44:30,900][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:44:31,402][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:44:31,905][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:44:32,418][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:44:32,918][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:44:33,417][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:44:33,921][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:44:34,422][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:44:34,925][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:44:35,425][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:44:35,928][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:44:36,435][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:44:36,935][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:44:37,441][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:44:37,943][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:44:38,445][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:44:38,947][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:44:39,443][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:44:39,943][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:44:40,441][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:44:40,943][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:44:41,442][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:44:41,942][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:44:42,441][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:44:42,941][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:44:43,440][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:44:43,942][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:44:44,443][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:44:44,945][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:44:45,449][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:44:45,945][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:44:46,444][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:44:46,944][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:44:47,441][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:44:47,941][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:44:48,441][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:44:48,939][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:44:49,439][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:44:49,942][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:44:50,444][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:44:50,988][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:44:51,485][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:44:51,982][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10110 tokens.
[2025-11-13 08:44:52,777][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 08:44:53,560][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:44:53,561][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:44:53,563][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:44:54,631][__main__][INFO] - Iteration 655 took 1m 6s (44.93% Gen, 53.46% Train). Generation: 29s, Training: 35s. Estimated remaining time: 44h 51m 43s. Estimated total time: 55h 26m 56s. Time estimates for 10 more iterations: 11m 5s, 100 more iterations: 1h 50m 53s, 500 more iterations: 9h 14m 29s.
[2025-11-13 08:44:54,633][__main__][INFO] - Starting iteration 655.
[2025-11-13 08:44:55,124][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:44:55,125][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:45:18,663][__main__][INFO] - Number of regex retries in iteration 655: 0
[2025-11-13 08:45:18,664][__main__][INFO] - agents played in iteration 655 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:45:19,515][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:19,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:19,583][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:19,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:19,608][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:45:19,610][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:45:20,427][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:45:20,898][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:45:21,404][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:45:21,906][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:45:22,421][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:45:22,926][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:45:23,430][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:45:23,934][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:45:24,437][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:45:24,941][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:45:25,441][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:45:25,945][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:45:26,756][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:45:28,179][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:45:28,737][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:45:29,247][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:45:29,749][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:45:30,251][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:45:30,753][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:45:31,254][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:45:31,762][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:45:32,262][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:45:32,764][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:45:33,269][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:45:33,769][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:45:34,271][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:45:34,773][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:45:35,276][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:45:35,780][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:45:36,285][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:45:36,790][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:45:37,291][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:45:37,796][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:45:38,311][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:45:38,811][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:45:39,312][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:45:39,815][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:45:40,315][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:45:40,826][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:45:41,327][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:45:41,831][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:45:42,332][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:45:42,832][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:45:43,335][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:45:43,837][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:45:44,342][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:45:44,841][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:45:45,339][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:45:45,835][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:45:46,330][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:45:46,828][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:45:47,331][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:45:47,824][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:45:48,320][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:45:48,824][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:45:49,323][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:45:49,821][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:45:50,319][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:45:50,819][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:45:51,319][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:45:51,817][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:45:52,318][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:45:52,831][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:45:53,328][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:45:53,826][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10107 tokens.
[2025-11-13 08:45:54,685][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:34
[2025-11-13 08:45:55,415][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:45:55,417][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:45:55,419][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:45:56,465][__main__][INFO] - Iteration 656 took 1m 1s (38.37% Gen, 59.92% Train). Generation: 23s, Training: 36s. Estimated remaining time: 40h 30m 48s. Estimated total time: 51h 7m 2s. Time estimates for 10 more iterations: 10m 13s, 100 more iterations: 1h 42m 14s, 500 more iterations: 8h 31m 10s.
[2025-11-13 08:45:56,467][__main__][INFO] - Starting iteration 656.
[2025-11-13 08:45:56,975][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:45:56,976][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:46:29,190][__main__][INFO] - Number of regex retries in iteration 656: 0
[2025-11-13 08:46:29,191][__main__][INFO] - agents played in iteration 656 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:46:30,049][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:30,078][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:30,105][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:30,129][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:30,129][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:46:30,130][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:46:31,037][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:46:31,494][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:46:32,006][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:46:32,508][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:46:33,014][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:46:33,524][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:46:34,025][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:46:34,532][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:46:35,036][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:46:35,541][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:46:36,045][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:46:36,551][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:46:37,058][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:46:37,563][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:46:38,066][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:46:38,571][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:46:39,074][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:46:39,573][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:46:40,076][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:46:40,577][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:46:41,080][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:46:41,584][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:46:42,082][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:46:42,586][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:46:43,089][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:46:43,587][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:46:44,086][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:46:44,586][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:46:45,089][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:46:45,600][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:46:46,101][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:46:46,605][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:46:47,106][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:46:47,606][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:46:48,122][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:46:48,621][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:46:49,123][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:46:49,623][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:46:50,121][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:46:50,629][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:46:51,128][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:46:51,626][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:46:52,131][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:46:52,633][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:46:53,129][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:46:53,629][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:46:54,128][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:46:54,633][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:46:55,128][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:46:55,628][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:46:56,130][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:46:56,631][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:46:57,132][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:46:57,631][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:46:58,131][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:46:58,641][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:46:59,142][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:46:59,643][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:47:00,147][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:47:00,644][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:47:01,144][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:47:01,645][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:47:02,147][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:47:02,657][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:47:03,157][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10179 tokens.
[2025-11-13 08:47:04,104][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.34%, ΔTime: 00:00:33
[2025-11-13 08:47:04,850][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:47:04,851][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:47:04,853][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:47:05,864][__main__][INFO] - Iteration 657 took 1m 8s (46.76% Gen, 51.77% Train). Generation: 32s, Training: 35s. Estimated remaining time: 46h 47m 5s. Estimated total time: 57h 24m 30s. Time estimates for 10 more iterations: 11m 28s, 100 more iterations: 1h 54m 49s, 500 more iterations: 9h 34m 5s.
[2025-11-13 08:47:05,866][__main__][INFO] - Starting iteration 657.
[2025-11-13 08:47:06,366][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:47:06,367][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:47:23,428][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:47:28,074][__main__][INFO] - Number of regex retries in iteration 657: 1
[2025-11-13 08:47:28,075][__main__][INFO] - agents played in iteration 657 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:47:28,955][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:28,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:29,006][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:29,030][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:29,030][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:47:29,031][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:47:29,933][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:47:30,391][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:47:30,897][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:47:31,397][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:47:31,896][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:47:32,400][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:47:32,910][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:47:33,410][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:47:33,909][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:47:34,416][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:47:34,918][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:47:35,433][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:47:35,935][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:47:36,438][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:47:36,948][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:47:37,453][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:47:37,959][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:47:38,463][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:47:38,962][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:47:39,463][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:47:39,963][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:47:40,463][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:47:40,965][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:47:41,467][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:47:41,968][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:47:42,471][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:47:42,971][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:47:43,471][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:47:43,971][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:47:44,475][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:47:44,978][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:47:45,482][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:47:45,982][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:47:46,485][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:47:46,984][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:47:47,486][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:47:47,988][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:47:48,488][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:47:48,992][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:47:49,493][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:47:49,991][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:47:50,497][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:47:50,998][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:47:51,498][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:47:51,998][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:47:52,501][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:47:53,007][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:47:53,511][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:47:54,014][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:47:54,521][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:47:55,020][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:47:55,534][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:47:56,033][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:47:56,531][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:47:57,028][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:47:57,527][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:47:58,024][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:47:58,520][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:47:59,016][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:47:59,513][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:48:00,008][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:48:00,507][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:48:01,016][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:48:01,513][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:48:02,010][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10010 tokens.
[2025-11-13 08:48:02,853][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.04%, ΔTime: 00:00:32
[2025-11-13 08:48:03,629][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:48:03,630][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:48:03,632][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:48:04,645][__main__][INFO] - Iteration 658 took 58s (37.25% Gen, 61.01% Train). Generation: 21s, Training: 35s. Estimated remaining time: 37h 55m 35s. Estimated total time: 48h 33m 58s. Time estimates for 10 more iterations: 9m 42s, 100 more iterations: 1h 37m 7s, 500 more iterations: 8h 5m 39s.
[2025-11-13 08:48:04,647][__main__][INFO] - Starting iteration 658.
[2025-11-13 08:48:05,133][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:48:05,134][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:48:36,934][__main__][INFO] - Number of regex retries in iteration 658: 0
[2025-11-13 08:48:36,935][__main__][INFO] - agents played in iteration 658 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:48:37,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:37,831][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:37,855][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:37,877][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:37,878][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:48:37,878][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:48:38,782][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:48:39,246][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:48:39,749][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:48:40,264][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:48:40,769][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:48:41,271][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:48:41,773][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:48:42,277][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:48:42,783][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:48:43,285][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:48:43,786][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:48:44,289][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:48:44,792][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:48:45,296][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:48:45,801][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:48:46,309][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:48:46,810][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:48:47,311][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:48:47,813][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:48:48,314][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:48:48,815][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:48:49,326][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:48:49,826][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:48:50,328][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:48:50,835][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:48:51,334][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:48:51,846][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:48:52,345][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:48:52,847][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:48:53,345][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:48:53,843][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:48:54,344][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:48:54,844][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:48:55,345][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:48:55,847][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:48:56,348][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:48:56,854][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:48:57,360][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:48:57,861][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:48:58,364][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:48:58,864][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:48:59,365][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:48:59,870][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:49:00,377][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:49:00,879][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:49:01,383][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:49:01,888][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:49:02,395][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:49:02,898][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:49:03,400][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:49:03,904][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:49:04,408][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:49:04,905][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:49:05,408][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:49:05,911][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:49:06,411][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:49:06,911][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:49:07,412][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:49:07,923][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:49:08,421][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:49:08,922][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:49:09,418][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:49:09,919][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:49:10,418][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:49:10,915][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10089 tokens.
[2025-11-13 08:49:11,759][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 08:49:12,514][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:49:12,516][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:49:12,518][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:49:13,689][__main__][INFO] - Iteration 659 took 1m 8s (46.39% Gen, 51.90% Train). Generation: 31s, Training: 35s. Estimated remaining time: 46h 28m 17s. Estimated total time: 57h 7m 49s. Time estimates for 10 more iterations: 11m 25s, 100 more iterations: 1h 54m 15s, 500 more iterations: 9h 31m 18s.
[2025-11-13 08:49:13,691][__main__][INFO] - Starting iteration 659.
[2025-11-13 08:49:14,188][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:49:14,189][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:49:42,496][__main__][INFO] - Number of regex retries in iteration 659: 0
[2025-11-13 08:49:42,497][__main__][INFO] - agents played in iteration 659 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:49:43,370][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:43,398][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:43,424][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:43,447][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:43,448][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:49:43,448][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:49:44,339][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:49:44,798][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:49:45,310][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:49:45,814][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:49:46,317][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:49:46,821][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:49:47,333][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:49:47,832][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:49:48,342][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:49:48,842][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:49:49,345][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:49:49,846][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:49:50,348][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:49:50,852][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:49:51,367][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:49:51,870][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:49:52,370][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:49:52,872][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:49:53,374][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:49:53,882][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:49:54,388][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:49:54,894][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:49:55,398][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:49:55,904][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:49:56,410][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:49:56,912][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:49:57,417][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:49:57,921][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:49:58,425][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:49:58,926][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:49:59,446][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:49:59,948][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:50:00,460][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:50:00,963][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:50:01,463][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:50:01,969][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:50:02,471][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:50:02,973][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:50:03,475][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:50:03,979][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:50:04,485][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:50:04,985][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:50:05,486][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:50:05,989][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:50:06,490][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:50:06,988][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:50:07,489][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:50:07,989][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:50:08,487][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:50:08,987][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:50:09,491][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:50:09,989][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:50:10,489][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:50:10,985][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:50:11,482][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:50:11,979][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:50:12,478][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:50:12,975][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:50:13,477][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:50:13,970][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:50:14,466][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:50:14,978][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:50:15,474][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:50:15,970][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:50:16,465][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10110 tokens.
[2025-11-13 08:50:17,342][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.04%, ΔTime: 00:00:33
[2025-11-13 08:50:18,096][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:50:18,097][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:50:18,099][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:50:19,100][__main__][INFO] - Iteration 660 took 1m 4s (43.61% Gen, 54.85% Train). Generation: 28s, Training: 35s. Estimated remaining time: 43h 25m 0s. Estimated total time: 54h 5m 37s. Time estimates for 10 more iterations: 10m 49s, 100 more iterations: 1h 48m 11s, 500 more iterations: 9h 0m 56s.
[2025-11-13 08:50:19,102][__main__][INFO] - Starting iteration 660.
[2025-11-13 08:50:19,594][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 65 and human policies 1.
[2025-11-13 08:50:19,595][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:50:46,035][__main__][INFO] - Number of regex retries in iteration 660: 0
[2025-11-13 08:50:46,035][__main__][INFO] - agents played in iteration 660 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:50:46,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:46,895][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:46,920][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:46,943][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:46,944][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:50:46,944][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:50:47,826][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:50:48,284][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:50:48,786][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:50:49,293][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:50:49,793][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:50:50,301][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:50:50,804][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:50:51,307][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:50:51,807][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:50:52,311][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:50:53,055][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:50:54,826][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:50:55,337][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:50:55,842][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:50:56,345][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:50:56,852][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:50:57,357][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:50:57,857][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:50:58,356][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:50:58,855][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:50:59,369][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:50:59,870][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:51:00,371][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:51:00,878][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:51:01,377][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:51:01,891][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:51:02,397][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:51:02,897][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:51:03,398][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:51:03,897][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:51:04,400][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:51:04,904][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:51:05,404][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:51:05,912][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:51:06,415][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:51:06,917][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:51:07,417][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:51:07,918][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:51:08,421][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:51:08,925][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:51:09,425][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:51:09,927][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:51:10,430][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:51:10,931][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:51:11,431][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:51:11,935][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:51:12,435][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:51:12,942][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:51:13,447][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:51:13,949][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:51:14,449][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:51:14,949][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:51:15,447][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:51:15,945][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:51:16,448][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:51:16,949][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:51:17,447][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:51:17,946][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:51:18,446][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:51:18,943][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:51:19,443][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:51:19,940][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:51:20,439][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:51:20,939][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:51:21,438][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10207 tokens.
[2025-11-13 08:51:22,286][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:34
[2025-11-13 08:51:23,004][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:51:23,006][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:51:23,008][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:51:24,937][__main__][INFO] - Iteration 661 took 1m 5s (40.46% Gen, 56.58% Train). Generation: 26s, Training: 36s. Estimated remaining time: 43h 45m 26s. Estimated total time: 54h 27m 10s. Time estimates for 10 more iterations: 10m 53s, 100 more iterations: 1h 48m 54s, 500 more iterations: 9h 4m 31s.
[2025-11-13 08:51:24,939][__main__][INFO] - Starting iteration 661.
[2025-11-13 08:51:25,422][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 08:51:25,422][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:51:49,359][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:52:01,291][__main__][INFO] - Number of regex retries in iteration 661: 1
[2025-11-13 08:52:01,291][__main__][INFO] - agents played in iteration 661 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:52:02,138][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:02,161][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:02,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:02,210][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:02,210][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:52:02,211][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:52:03,112][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:52:03,575][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:52:04,081][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:52:04,585][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:52:05,109][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:52:05,616][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:52:06,121][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:52:06,624][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:52:07,128][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:52:07,631][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:52:08,132][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:52:08,636][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:52:09,144][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:52:09,645][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:52:10,147][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:52:10,651][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:52:11,153][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:52:11,656][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:52:12,158][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:52:12,663][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:52:13,164][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:52:13,665][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:52:14,166][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:52:14,670][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:52:15,169][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:52:15,678][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:52:16,177][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:52:16,679][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:52:17,183][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:52:17,683][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:52:18,185][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:52:18,698][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:52:19,199][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:52:19,715][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:52:20,217][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:52:20,714][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:52:21,216][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:52:21,719][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:52:22,224][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:52:22,726][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:52:23,222][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:52:23,720][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:52:24,219][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:52:24,717][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:52:25,221][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:52:25,724][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:52:26,228][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:52:26,726][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:52:27,223][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:52:27,724][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:52:28,219][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:52:28,717][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:52:29,218][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:52:29,715][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:52:30,229][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:52:30,727][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:52:31,228][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:52:31,726][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:52:32,224][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:52:32,730][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:52:33,230][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:52:33,732][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:52:34,237][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:52:34,737][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:52:35,243][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10165 tokens.
[2025-11-13 08:52:36,084][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 08:52:36,857][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:52:36,859][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:52:36,861][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:52:37,873][__main__][INFO] - Iteration 662 took 1m 12s (49.51% Gen, 49.09% Train). Generation: 35s, Training: 35s. Estimated remaining time: 49h 39m 39s. Estimated total time: 60h 22m 35s. Time estimates for 10 more iterations: 12m 4s, 100 more iterations: 2h 0m 45s, 500 more iterations: 10h 3m 45s.
[2025-11-13 08:52:37,875][__main__][INFO] - Starting iteration 662.
[2025-11-13 08:52:38,368][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 08:52:38,369][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:52:59,259][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 08:53:07,446][__main__][INFO] - Number of regex retries in iteration 662: 1
[2025-11-13 08:53:07,446][__main__][INFO] - agents played in iteration 662 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:53:08,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:08,292][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:08,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:08,338][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:08,339][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:53:08,339][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:53:09,241][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:53:09,713][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:53:10,225][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:53:10,728][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:53:11,241][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:53:11,749][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:53:12,252][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:53:12,758][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:53:13,260][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:53:13,764][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:53:14,267][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:53:14,770][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:53:15,271][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:53:15,774][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:53:16,279][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:53:16,787][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:53:17,290][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:53:17,799][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:53:18,303][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:53:18,804][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:53:19,306][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:53:19,807][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:53:20,319][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:53:20,823][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:53:21,324][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:53:21,823][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:53:22,322][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:53:22,821][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:53:23,320][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:53:23,820][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:53:24,318][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:53:24,820][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:53:25,322][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:53:25,832][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:53:26,336][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:53:26,839][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:53:27,337][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:53:27,839][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:53:28,343][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:53:28,841][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:53:29,344][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:53:29,839][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:53:30,336][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:53:30,844][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:53:31,341][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:53:31,839][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:53:32,335][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:53:32,835][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:53:33,338][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:53:33,841][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:53:34,342][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:53:34,844][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:53:35,345][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:53:35,841][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:53:36,341][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:53:36,836][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:53:37,336][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:53:37,834][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:53:38,330][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:53:38,826][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:53:39,322][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:53:39,820][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:53:40,318][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:53:40,813][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:53:41,313][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10177 tokens.
[2025-11-13 08:53:42,126][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.24%, Current % of VRAM taken: 60.50%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:32
[2025-11-13 08:53:42,877][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:53:42,878][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:53:42,880][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:53:44,020][__main__][INFO] - Iteration 663 took 1m 5s (44.29% Gen, 53.97% Train). Generation: 29s, Training: 35s. Estimated remaining time: 43h 58m 35s. Estimated total time: 54h 42m 37s. Time estimates for 10 more iterations: 10m 56s, 100 more iterations: 1h 49m 25s, 500 more iterations: 9h 7m 6s.
[2025-11-13 08:53:44,022][__main__][INFO] - Starting iteration 663.
[2025-11-13 08:53:44,491][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 08:53:44,491][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:54:04,232][__main__][INFO] - Number of regex retries in iteration 663: 0
[2025-11-13 08:54:04,233][__main__][INFO] - agents played in iteration 663 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:54:05,048][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:05,071][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:05,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:05,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:05,120][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:54:05,120][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:54:05,957][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:54:06,434][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:54:06,939][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:54:07,442][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:54:07,942][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:54:08,442][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:54:08,949][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:54:09,451][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:54:09,956][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:54:10,465][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:54:10,968][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:54:11,470][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:54:11,973][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:54:12,475][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:54:12,984][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:54:13,490][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:54:13,992][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:54:14,663][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:54:15,925][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:54:16,423][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:54:16,945][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:54:17,445][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:54:17,951][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:54:18,452][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:54:18,952][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:54:19,452][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:54:19,956][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:54:20,456][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:54:20,958][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:54:21,457][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:54:21,956][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:54:22,460][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:54:22,958][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:54:23,460][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:54:23,966][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:54:24,466][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:54:24,986][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:54:25,491][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:54:25,991][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:54:26,490][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:54:26,988][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:54:27,486][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:54:27,983][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:54:28,480][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:54:28,983][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:54:29,481][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:54:29,982][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:54:30,480][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:54:30,977][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:54:31,484][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:54:31,985][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:54:32,487][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:54:32,991][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:54:33,493][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:54:33,993][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:54:34,493][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:54:34,992][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:54:35,494][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:54:35,991][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:54:36,489][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:54:36,988][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:54:37,486][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:54:37,989][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:54:38,486][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:54:38,985][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10049 tokens.
[2025-11-13 08:54:39,840][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:33
[2025-11-13 08:54:40,579][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:54:40,581][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:54:40,582][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:54:41,593][__main__][INFO] - Iteration 664 took 57s (34.57% Gen, 63.66% Train). Generation: 19s, Training: 36s. Estimated remaining time: 36h 50m 7s. Estimated total time: 47h 35m 7s. Time estimates for 10 more iterations: 9m 31s, 100 more iterations: 1h 35m 10s, 500 more iterations: 7h 55m 51s.
[2025-11-13 08:54:41,595][__main__][INFO] - Starting iteration 664.
[2025-11-13 08:54:42,089][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 08:54:42,089][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:55:11,385][__main__][INFO] - Number of regex retries in iteration 664: 0
[2025-11-13 08:55:11,386][__main__][INFO] - agents played in iteration 664 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:55:12,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:12,355][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:12,378][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:12,401][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:12,402][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:55:12,404][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:55:13,298][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:55:13,756][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:55:14,263][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:55:14,768][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:55:15,274][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:55:15,776][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:55:16,284][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:55:16,785][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:55:17,286][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:55:17,793][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:55:18,294][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:55:18,801][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:55:19,303][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:55:19,802][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:55:20,309][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:55:20,811][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:55:21,311][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:55:21,812][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:55:22,316][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:55:22,820][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:55:23,330][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:55:23,830][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:55:24,330][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:55:24,834][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:55:25,334][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:55:25,835][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:55:26,338][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:55:26,839][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:55:27,343][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:55:27,846][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:55:28,348][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:55:28,850][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:55:29,380][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:55:29,881][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:55:30,384][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:55:30,888][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:55:31,392][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:55:31,892][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:55:32,395][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:55:32,895][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:55:33,395][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:55:33,892][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:55:34,388][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:55:34,885][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:55:35,382][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:55:35,880][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:55:36,378][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:55:36,875][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:55:37,370][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:55:37,870][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:55:38,371][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:55:38,878][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:55:39,378][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:55:39,883][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:55:40,384][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:55:40,886][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:55:41,386][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:55:41,886][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:55:42,386][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:55:42,885][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:55:43,382][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:55:43,878][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:55:44,373][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:55:44,869][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:55:45,364][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10086 tokens.
[2025-11-13 08:55:46,189][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.37%, ΔTime: 00:00:32
[2025-11-13 08:55:46,963][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:55:46,965][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:55:46,966][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:55:48,074][__main__][INFO] - Iteration 665 took 1m 5s (44.40% Gen, 53.92% Train). Generation: 29s, Training: 35s. Estimated remaining time: 44h 13m 12s. Estimated total time: 54h 59m 18s. Time estimates for 10 more iterations: 10m 59s, 100 more iterations: 1h 49m 58s, 500 more iterations: 9h 9m 53s.
[2025-11-13 08:55:48,077][__main__][INFO] - Starting iteration 665.
[2025-11-13 08:55:48,547][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 08:55:48,548][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:56:12,188][__main__][INFO] - Number of regex retries in iteration 665: 0
[2025-11-13 08:56:12,189][__main__][INFO] - agents played in iteration 665 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:56:12,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:13,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:13,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:13,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:13,063][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:56:13,064][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:56:13,953][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:56:14,412][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:56:14,926][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:56:15,430][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:56:15,932][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:56:16,454][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:56:16,956][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:56:17,459][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:56:17,962][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:56:18,468][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:56:18,970][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:56:19,473][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:56:19,975][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:56:20,484][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:56:20,986][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:56:21,488][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:56:21,990][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:56:22,491][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:56:22,994][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:56:23,494][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:56:23,994][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:56:24,495][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:56:24,997][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:56:25,497][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:56:26,000][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:56:26,504][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:56:27,005][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:56:27,505][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:56:28,004][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:56:28,504][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:56:29,002][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:56:29,506][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:56:30,003][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:56:30,502][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:56:31,002][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:56:31,503][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:56:32,005][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:56:32,509][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:56:33,009][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:56:33,507][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:56:34,006][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:56:34,503][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:56:35,003][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:56:35,505][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:56:36,002][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:56:36,500][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:56:37,001][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:56:37,500][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:56:38,002][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:56:38,498][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:56:38,997][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:56:39,498][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:56:39,996][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:56:40,494][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:56:40,994][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:56:41,495][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:56:41,993][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:56:42,488][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:56:42,981][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:56:43,477][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:56:43,974][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:56:44,469][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:56:44,969][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:56:45,475][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:56:45,970][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10144 tokens.
[2025-11-13 08:56:46,803][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:32
[2025-11-13 08:56:47,749][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:56:47,750][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:56:47,773][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:56:49,071][__main__][INFO] - Iteration 666 took 1m 0s (39.06% Gen, 58.80% Train). Generation: 23s, Training: 35s. Estimated remaining time: 39h 39m 5s. Estimated total time: 50h 26m 13s. Time estimates for 10 more iterations: 10m 5s, 100 more iterations: 1h 40m 52s, 500 more iterations: 8h 24m 22s.
[2025-11-13 08:56:49,073][__main__][INFO] - Starting iteration 666.
[2025-11-13 08:56:49,562][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 08:56:49,562][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:57:15,765][__main__][INFO] - Number of regex retries in iteration 666: 0
[2025-11-13 08:57:15,766][__main__][INFO] - agents played in iteration 666 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:57:16,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:16,844][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:16,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:16,905][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:16,906][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:57:16,908][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:57:17,850][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:57:18,315][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:57:18,817][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:57:19,319][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:57:19,831][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:57:20,333][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:57:20,838][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:57:21,339][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:57:21,844][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:57:22,348][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:57:22,851][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:57:23,352][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:57:23,861][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:57:24,364][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:57:24,884][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:57:25,386][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:57:25,888][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:57:26,400][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:57:26,904][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:57:27,407][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:57:27,910][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:57:28,413][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:57:28,915][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:57:29,419][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:57:29,924][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:57:30,424][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:57:30,924][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:57:31,428][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:57:31,927][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:57:32,427][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:57:32,930][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:57:33,426][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:57:33,927][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:57:34,426][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:57:34,931][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:57:35,433][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:57:35,932][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:57:36,430][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:57:36,929][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:57:37,429][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:57:37,933][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:57:38,438][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:57:38,943][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:57:39,456][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:57:39,956][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:57:40,466][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:57:40,967][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:57:41,467][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:57:41,966][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:57:42,463][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:57:42,960][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:57:43,459][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:57:43,960][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:57:44,461][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:57:44,961][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:57:45,457][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:57:45,969][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:57:46,474][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:57:46,972][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:57:47,470][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:57:47,967][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:57:48,465][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:57:48,964][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:57:49,462][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:57:49,960][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10090 tokens.
[2025-11-13 08:57:50,779][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:32
[2025-11-13 08:57:51,557][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:57:51,558][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:57:51,561][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:57:52,601][__main__][INFO] - Iteration 667 took 1m 3s (41.57% Gen, 56.78% Train). Generation: 26s, Training: 35s. Estimated remaining time: 41h 43m 47s. Estimated total time: 52h 31m 58s. Time estimates for 10 more iterations: 10m 30s, 100 more iterations: 1h 45m 3s, 500 more iterations: 8h 45m 19s.
[2025-11-13 08:57:52,602][__main__][INFO] - Starting iteration 667.
[2025-11-13 08:57:53,078][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 08:57:53,078][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:58:16,822][__main__][INFO] - Number of regex retries in iteration 667: 0
[2025-11-13 08:58:16,823][__main__][INFO] - agents played in iteration 667 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:58:17,635][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:17,666][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:17,693][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:17,717][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:17,718][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:58:17,719][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:58:18,634][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:58:19,117][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:58:19,627][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:58:20,130][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:58:20,630][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:58:21,131][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:58:21,638][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:58:22,144][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:58:22,647][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:58:23,150][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:58:23,653][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:58:24,175][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:58:24,680][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:58:25,186][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:58:25,689][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:58:26,191][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:58:26,704][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:58:27,206][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:58:27,708][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:58:28,212][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:58:28,716][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:58:29,218][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:58:29,718][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:58:30,217][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:58:30,724][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:58:31,225][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:58:31,729][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:58:32,231][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:58:32,735][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:58:33,243][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:58:33,744][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:58:34,243][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:58:34,752][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:58:35,252][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:58:35,752][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:58:36,256][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:58:36,759][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:58:37,272][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:58:37,771][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:58:38,271][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:58:38,782][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:58:39,281][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:58:39,802][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:58:40,302][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:58:40,801][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:58:41,301][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:58:41,801][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:58:42,305][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:58:42,804][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:58:43,303][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:58:43,807][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:58:44,309][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:58:44,806][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:58:45,306][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:58:45,806][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:58:46,309][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:58:46,809][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:58:47,312][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:58:47,819][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:58:48,320][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:58:48,825][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:58:49,328][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:58:49,828][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:58:50,328][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:58:50,826][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10138 tokens.
[2025-11-13 08:58:51,648][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:33
[2025-11-13 08:58:52,410][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:58:52,412][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:58:52,414][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:58:53,442][__main__][INFO] - Iteration 668 took 1m 0s (39.33% Gen, 58.96% Train). Generation: 23s, Training: 35s. Estimated remaining time: 39h 29m 3s. Estimated total time: 50h 18m 15s. Time estimates for 10 more iterations: 10m 3s, 100 more iterations: 1h 40m 36s, 500 more iterations: 8h 23m 2s.
[2025-11-13 08:58:53,444][__main__][INFO] - Starting iteration 668.
[2025-11-13 08:58:53,937][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 08:58:53,938][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:59:21,807][__main__][INFO] - Number of regex retries in iteration 668: 0
[2025-11-13 08:59:21,808][__main__][INFO] - agents played in iteration 668 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 08:59:22,714][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:22,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:22,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:22,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:22,806][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:59:22,807][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:59:23,739][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 08:59:24,200][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 08:59:24,716][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 08:59:25,218][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 08:59:25,719][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 08:59:26,224][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 08:59:26,725][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 08:59:27,227][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 08:59:27,727][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 08:59:28,229][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 08:59:28,732][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 08:59:29,232][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 08:59:29,741][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 08:59:30,244][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 08:59:30,747][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 08:59:31,256][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 08:59:31,758][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 08:59:32,263][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 08:59:32,766][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 08:59:33,268][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 08:59:33,771][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 08:59:34,272][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 08:59:34,774][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 08:59:35,293][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 08:59:35,795][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 08:59:36,304][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 08:59:36,806][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 08:59:37,318][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 08:59:37,824][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 08:59:38,326][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 08:59:38,828][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 08:59:39,335][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 08:59:39,837][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 08:59:40,336][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 08:59:40,836][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 08:59:41,335][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 08:59:41,838][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 08:59:42,340][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 08:59:42,842][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 08:59:43,341][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 08:59:43,843][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 08:59:44,349][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 08:59:44,856][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 08:59:45,361][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 08:59:45,867][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 08:59:46,368][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 08:59:46,869][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 08:59:47,371][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 08:59:47,873][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 08:59:48,371][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 08:59:48,867][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 08:59:49,364][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 08:59:49,871][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 08:59:50,368][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 08:59:50,867][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 08:59:51,365][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 08:59:51,864][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 08:59:52,363][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 08:59:52,861][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 08:59:53,360][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 08:59:53,863][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 08:59:54,362][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 08:59:54,862][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 08:59:55,362][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 08:59:55,861][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10120 tokens.
[2025-11-13 08:59:56,721][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-13 08:59:57,504][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:59:57,506][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:59:57,507][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:59:58,512][__main__][INFO] - Iteration 669 took 1m 4s (43.16% Gen, 55.28% Train). Generation: 27s, Training: 35s. Estimated remaining time: 42h 58m 30s. Estimated total time: 53h 48m 47s. Time estimates for 10 more iterations: 10m 45s, 100 more iterations: 1h 47m 37s, 500 more iterations: 8h 58m 7s.
[2025-11-13 08:59:58,514][__main__][INFO] - Starting iteration 669.
[2025-11-13 08:59:59,001][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 08:59:59,002][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:00:18,711][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:00:26,546][__main__][INFO] - Number of regex retries in iteration 669: 1
[2025-11-13 09:00:26,547][__main__][INFO] - agents played in iteration 669 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:00:27,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:27,391][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:27,424][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:27,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:27,454][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:00:27,455][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:00:28,245][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:00:28,702][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:00:29,201][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:00:29,698][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:00:30,206][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:00:30,701][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:00:31,198][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:00:31,705][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:00:32,210][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:00:32,711][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:00:33,212][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:00:33,722][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:00:34,227][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:00:34,729][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:00:35,232][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:00:35,736][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:00:36,242][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:00:36,746][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:00:37,247][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:00:37,753][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:00:38,255][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:00:38,758][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:00:39,261][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:00:39,763][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:00:40,266][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:00:40,771][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:00:41,273][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:00:41,776][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:00:42,278][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:00:42,781][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:00:43,281][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:00:43,783][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:00:44,286][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:00:44,795][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:00:45,297][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:00:45,804][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:00:46,311][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:00:46,812][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:00:47,322][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:00:47,826][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:00:48,325][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:00:48,827][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:00:49,328][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:00:49,825][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:00:50,323][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:00:50,820][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:00:51,332][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:00:51,830][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:00:52,330][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:00:52,827][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:00:53,327][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:00:53,824][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:00:54,322][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:00:54,823][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:00:55,322][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:00:55,824][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:00:56,322][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:00:56,824][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:00:57,323][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:00:57,832][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:00:58,334][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:00:58,833][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:00:59,338][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:00:59,844][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:01:00,349][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10095 tokens.
[2025-11-13 09:01:01,198][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.08%, ΔTime: 00:00:32
[2025-11-13 09:01:01,978][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:01:01,979][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:01:01,980][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:01:03,157][__main__][INFO] - Iteration 670 took 1m 4s (42.93% Gen, 55.23% Train). Generation: 27s, Training: 35s. Estimated remaining time: 42h 36m 28s. Estimated total time: 53h 27m 49s. Time estimates for 10 more iterations: 10m 41s, 100 more iterations: 1h 46m 55s, 500 more iterations: 8h 54m 38s.
[2025-11-13 09:01:03,159][__main__][INFO] - Starting iteration 670.
[2025-11-13 09:01:03,643][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 66 and human policies 1.
[2025-11-13 09:01:03,644][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:01:14,563][mllm.models.large_language_model_local][WARNING] - Response Proposal: x hats, y books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:01:24,717][__main__][INFO] - Number of regex retries in iteration 670: 1
[2025-11-13 09:01:24,718][__main__][INFO] - agents played in iteration 670 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:01:25,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:25,513][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:25,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:25,562][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:25,562][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:01:25,563][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:01:26,365][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:01:26,843][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:01:27,342][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:01:27,838][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:01:28,337][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:01:28,837][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:01:29,350][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:01:29,848][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:01:30,347][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:01:30,843][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:01:31,344][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:01:31,847][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:01:32,347][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:01:32,851][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:01:33,354][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:01:33,858][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:01:34,365][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:01:34,868][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:01:35,372][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:01:35,875][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:01:36,379][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:01:36,881][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:01:37,382][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:01:37,886][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:01:38,390][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:01:38,891][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:01:39,391][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:01:39,893][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:01:40,444][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:01:42,238][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:01:42,764][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:01:43,267][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:01:43,771][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:01:44,275][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:01:44,775][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:01:45,277][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:01:45,778][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:01:46,279][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:01:46,779][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:01:47,282][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:01:47,784][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:01:48,285][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:01:48,787][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:01:49,292][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:01:49,793][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:01:50,295][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:01:50,801][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:01:51,302][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:01:51,803][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:01:52,309][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:01:52,812][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:01:53,316][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:01:53,818][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:01:54,327][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:01:54,826][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:01:55,338][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:01:55,837][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:01:56,338][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:01:56,834][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:01:57,330][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:01:57,830][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:01:58,329][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:01:58,827][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:01:59,341][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:01:59,838][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10094 tokens.
[2025-11-13 09:02:00,736][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:34
[2025-11-13 09:02:01,374][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:02:01,375][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:02:01,377][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:02:03,404][__main__][INFO] - Iteration 671 took 59s (35.26% Gen, 61.34% Train). Generation: 21s, Training: 36s. Estimated remaining time: 38h 55m 41s. Estimated total time: 49h 48m 2s. Time estimates for 10 more iterations: 9m 57s, 100 more iterations: 1h 39m 36s, 500 more iterations: 8h 18m 0s.
[2025-11-13 09:02:03,407][__main__][INFO] - Starting iteration 671.
[2025-11-13 09:02:03,886][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:02:03,887][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:02:31,190][__main__][INFO] - Number of regex retries in iteration 671: 0
[2025-11-13 09:02:31,191][__main__][INFO] - agents played in iteration 671 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:02:32,090][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:32,112][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:32,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:32,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:32,157][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:02:32,158][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:02:32,955][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:02:33,408][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:02:33,907][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:02:34,408][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:02:34,904][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:02:35,402][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:02:35,901][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:02:36,398][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:02:36,897][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:02:37,398][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:02:37,896][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:02:38,394][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:02:38,896][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:02:39,395][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:02:39,891][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:02:40,388][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:02:40,885][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:02:41,384][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:02:41,885][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:02:42,388][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:02:42,888][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:02:43,389][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:02:43,894][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:02:44,395][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:02:44,897][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:02:45,399][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:02:45,903][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:02:46,404][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:02:46,911][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:02:47,411][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:02:47,925][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:02:48,427][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:02:48,940][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:02:49,444][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:02:49,946][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:02:50,450][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:02:50,950][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:02:51,454][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:02:51,962][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:02:52,465][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:02:52,969][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:02:53,472][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:02:53,976][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:02:54,478][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:02:54,981][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:02:55,486][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:02:55,988][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:02:56,488][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:02:56,998][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:02:57,497][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:02:57,997][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:02:58,500][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:02:59,002][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:02:59,506][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:03:00,006][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:03:00,508][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:03:01,015][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:03:01,515][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:03:02,025][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:03:02,530][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:03:03,031][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:03:03,565][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:03:04,067][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:03:04,571][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:03:05,076][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10169 tokens.
[2025-11-13 09:03:05,962][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:33
[2025-11-13 09:03:06,731][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:03:06,733][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:03:06,735][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:03:07,814][__main__][INFO] - Iteration 672 took 1m 3s (42.71% Gen, 55.60% Train). Generation: 27s, Training: 35s. Estimated remaining time: 42h 23m 0s. Estimated total time: 53h 16m 26s. Time estimates for 10 more iterations: 10m 39s, 100 more iterations: 1h 46m 32s, 500 more iterations: 8h 52m 44s.
[2025-11-13 09:03:07,816][__main__][INFO] - Starting iteration 672.
[2025-11-13 09:03:08,299][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:03:08,299][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:03:35,926][__main__][INFO] - Number of regex retries in iteration 672: 0
[2025-11-13 09:03:35,926][__main__][INFO] - agents played in iteration 672 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:03:36,765][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:36,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:36,815][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:36,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:36,838][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:03:36,839][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:03:37,652][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:03:38,109][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:03:38,614][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:03:39,112][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:03:39,612][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:03:40,113][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:03:40,610][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:03:41,108][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:03:41,606][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:03:42,106][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:03:42,602][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:03:43,105][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:03:43,601][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:03:44,098][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:03:44,599][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:03:45,096][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:03:45,592][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:03:46,093][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:03:46,593][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:03:47,100][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:03:47,603][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:03:48,105][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:03:48,609][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:03:49,117][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:03:49,619][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:03:50,120][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:03:50,624][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:03:51,139][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:03:51,646][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:03:52,150][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:03:52,650][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:03:53,150][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:03:53,666][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:03:54,169][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:03:54,671][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:03:55,174][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:03:55,676][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:03:56,179][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:03:56,684][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:03:57,190][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:03:57,693][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:03:58,193][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:03:58,697][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:03:59,199][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:03:59,700][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:04:00,200][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:04:00,706][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:04:01,205][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:04:01,709][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:04:02,214][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:04:02,712][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:04:03,217][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:04:03,717][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:04:04,217][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:04:04,719][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:04:05,216][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:04:05,719][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:04:06,217][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:04:06,720][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:04:07,223][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:04:07,722][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:04:08,227][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:04:08,726][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:04:09,230][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:04:09,741][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10186 tokens.
[2025-11-13 09:04:10,634][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:32
[2025-11-13 09:04:11,378][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:04:11,380][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:04:11,381][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:04:12,467][__main__][INFO] - Iteration 673 took 1m 4s (43.05% Gen, 55.25% Train). Generation: 27s, Training: 35s. Estimated remaining time: 42h 33m 54s. Estimated total time: 53h 28m 25s. Time estimates for 10 more iterations: 10m 41s, 100 more iterations: 1h 46m 56s, 500 more iterations: 8h 54m 44s.
[2025-11-13 09:04:12,469][__main__][INFO] - Starting iteration 673.
[2025-11-13 09:04:12,952][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:04:12,952][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:04:35,513][__main__][INFO] - Number of regex retries in iteration 673: 0
[2025-11-13 09:04:35,513][__main__][INFO] - agents played in iteration 673 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:04:36,280][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:36,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:36,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:36,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:36,349][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:04:36,350][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:04:37,166][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:04:37,625][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:04:38,125][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:04:38,626][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:04:39,123][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:04:39,621][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:04:40,118][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:04:40,620][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:04:41,118][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:04:41,618][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:04:42,113][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:04:42,611][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:04:43,106][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:04:43,603][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:04:44,100][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:04:44,595][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:04:45,092][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:04:45,591][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:04:46,090][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:04:46,592][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:04:47,093][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:04:47,594][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:04:48,092][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:04:48,593][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:04:49,098][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:04:49,603][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:04:50,110][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:04:50,610][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:04:51,114][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:04:51,853][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:04:52,524][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:04:53,026][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:04:53,528][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:04:54,029][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:04:54,530][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:04:55,031][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:04:55,545][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:04:56,047][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:04:56,555][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:04:57,062][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:04:57,564][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:04:58,080][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:04:58,589][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:04:59,096][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:04:59,596][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:05:00,097][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:05:00,600][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:05:01,105][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:05:01,614][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:05:02,116][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:05:02,617][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:05:03,119][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:05:03,622][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:05:04,123][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:05:04,630][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:05:05,133][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:05:05,638][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:05:06,150][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:05:06,652][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:05:07,160][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:05:07,667][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:05:08,176][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:05:08,680][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:05:09,183][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:05:09,694][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10245 tokens.
[2025-11-13 09:05:10,642][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.26%, Current % of VRAM taken: 60.52%, Block Peak % of device VRAM: 62.26%, ΔTime: 00:00:33
[2025-11-13 09:05:11,379][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:05:11,381][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:05:11,383][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:05:12,443][__main__][INFO] - Iteration 674 took 59s (37.92% Gen, 60.29% Train). Generation: 22s, Training: 35s. Estimated remaining time: 38h 39m 4s. Estimated total time: 49h 34m 35s. Time estimates for 10 more iterations: 9m 54s, 100 more iterations: 1h 39m 9s, 500 more iterations: 8h 15m 45s.
[2025-11-13 09:05:12,445][__main__][INFO] - Starting iteration 674.
[2025-11-13 09:05:12,928][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:05:12,928][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:05:45,898][__main__][INFO] - Number of regex retries in iteration 674: 0
[2025-11-13 09:05:45,899][__main__][INFO] - agents played in iteration 674 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:05:46,707][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:46,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:46,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:46,778][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:46,779][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:05:46,780][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:05:47,627][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:05:48,081][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:05:48,584][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:05:49,097][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:05:49,596][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:05:50,097][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:05:50,594][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:05:51,090][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:05:51,591][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:05:52,087][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:05:52,585][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:05:53,094][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:05:53,590][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:05:54,087][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:05:54,584][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:05:55,082][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:05:55,597][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:05:56,101][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:05:56,603][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:05:57,104][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:05:57,612][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:05:58,117][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:05:58,615][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:05:59,119][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:05:59,623][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:06:00,127][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:06:00,631][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:06:01,134][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:06:01,634][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:06:02,135][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:06:02,644][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:06:03,146][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:06:03,650][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:06:04,153][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:06:04,681][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:06:05,187][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:06:05,693][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:06:06,207][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:06:06,711][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:06:07,215][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:06:07,721][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:06:08,232][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:06:08,741][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:06:09,247][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:06:09,752][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:06:10,277][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:06:10,783][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:06:11,289][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:06:11,795][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:06:12,302][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:06:12,819][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:06:13,325][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:06:13,829][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:06:14,339][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:06:14,848][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:06:15,354][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:06:15,858][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:06:16,367][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:06:16,869][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:06:17,371][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:06:17,878][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:06:18,380][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:06:18,889][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:06:19,404][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:06:19,907][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10105 tokens.
[2025-11-13 09:06:20,831][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:33
[2025-11-13 09:06:21,599][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:06:21,600][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:06:21,603][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:06:22,627][__main__][INFO] - Iteration 675 took 1m 9s (47.30% Gen, 51.23% Train). Generation: 32s, Training: 35s. Estimated remaining time: 47h 8m 18s. Estimated total time: 58h 4m 59s. Time estimates for 10 more iterations: 11m 36s, 100 more iterations: 1h 56m 9s, 500 more iterations: 9h 40m 49s.
[2025-11-13 09:06:22,629][__main__][INFO] - Starting iteration 675.
[2025-11-13 09:06:23,110][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:06:23,111][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:06:53,086][__main__][INFO] - Number of regex retries in iteration 675: 0
[2025-11-13 09:06:53,086][__main__][INFO] - agents played in iteration 675 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:06:53,853][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:53,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:53,901][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:53,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:53,924][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:06:53,925][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:06:54,720][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:06:55,172][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:06:55,675][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:06:56,193][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:06:56,692][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:06:57,187][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:06:57,689][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:06:58,190][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:06:58,699][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:06:59,196][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:06:59,692][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:07:00,198][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:07:00,699][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:07:01,198][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:07:01,696][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:07:02,191][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:07:02,687][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:07:03,186][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:07:03,683][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:07:04,197][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:07:04,694][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:07:05,200][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:07:05,703][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:07:06,208][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:07:06,714][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:07:07,215][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:07:07,714][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:07:08,220][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:07:08,719][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:07:09,223][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:07:09,724][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:07:10,224][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:07:10,730][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:07:11,231][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:07:11,734][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:07:12,248][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:07:13,652][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:07:14,217][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:07:14,738][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:07:15,256][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:07:15,756][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:07:16,260][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:07:16,761][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:07:17,270][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:07:17,773][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:07:18,276][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:07:18,788][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:07:19,292][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:07:19,806][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:07:20,306][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:07:20,806][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:07:21,309][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:07:21,811][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:07:22,314][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:07:22,813][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:07:23,314][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:07:23,823][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:07:24,324][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:07:24,825][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:07:25,330][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:07:25,830][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:07:26,335][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:07:26,835][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:07:27,338][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:07:27,845][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10153 tokens.
[2025-11-13 09:07:28,814][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:34
[2025-11-13 09:07:29,481][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:07:29,483][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:07:29,485][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:07:30,496][__main__][INFO] - Iteration 676 took 1m 7s (44.48% Gen, 54.01% Train). Generation: 29s, Training: 36s. Estimated remaining time: 45h 11m 33s. Estimated total time: 56h 9m 22s. Time estimates for 10 more iterations: 11m 13s, 100 more iterations: 1h 52m 18s, 500 more iterations: 9h 21m 33s.
[2025-11-13 09:07:30,500][__main__][INFO] - Starting iteration 676.
[2025-11-13 09:07:30,981][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:07:30,981][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:08:00,601][__main__][INFO] - Number of regex retries in iteration 676: 0
[2025-11-13 09:08:00,602][__main__][INFO] - agents played in iteration 676 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:08:01,522][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:01,549][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:01,576][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:01,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:01,600][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:08:01,601][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:08:02,409][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:08:02,866][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:08:03,366][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:08:03,865][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:08:04,362][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:08:04,856][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:08:05,355][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:08:05,851][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:08:06,347][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:08:06,843][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:08:07,338][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:08:07,833][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:08:08,330][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:08:08,825][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:08:09,319][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:08:09,814][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:08:10,308][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:08:10,804][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:08:11,305][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:08:11,805][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:08:12,315][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:08:12,814][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:08:13,313][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:08:13,814][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:08:14,314][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:08:14,815][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:08:15,320][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:08:15,825][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:08:16,334][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:08:16,846][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:08:17,349][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:08:17,857][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:08:18,358][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:08:18,861][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:08:19,364][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:08:19,869][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:08:20,373][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:08:20,875][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:08:21,379][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:08:21,880][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:08:22,382][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:08:22,889][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:08:23,388][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:08:23,889][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:08:24,397][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:08:24,899][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:08:25,409][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:08:25,913][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:08:26,413][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:08:26,916][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:08:27,419][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:08:27,922][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:08:28,423][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:08:28,925][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:08:29,432][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:08:29,932][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:08:30,431][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:08:30,933][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:08:31,432][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:08:31,932][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:08:32,432][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:08:32,930][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:08:33,440][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:08:33,942][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:08:34,442][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10072 tokens.
[2025-11-13 09:08:35,364][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:32
[2025-11-13 09:08:36,133][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:08:36,135][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:08:36,136][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:08:37,184][__main__][INFO] - Iteration 677 took 1m 6s (44.74% Gen, 53.67% Train). Generation: 29s, Training: 35s. Estimated remaining time: 44h 11m 18s. Estimated total time: 55h 10m 13s. Time estimates for 10 more iterations: 11m 2s, 100 more iterations: 1h 50m 20s, 500 more iterations: 9h 11m 42s.
[2025-11-13 09:08:37,186][__main__][INFO] - Starting iteration 677.
[2025-11-13 09:08:37,672][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:08:37,672][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:09:04,721][__main__][INFO] - Number of regex retries in iteration 677: 0
[2025-11-13 09:09:04,721][__main__][INFO] - agents played in iteration 677 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:09:05,482][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:05,507][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:05,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:05,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:05,554][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:09:05,556][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:09:06,358][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:09:06,809][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:09:07,335][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:09:07,834][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:09:08,332][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:09:08,828][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:09:09,325][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:09:09,833][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:09:10,329][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:09:10,827][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:09:11,341][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:09:11,840][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:09:12,343][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:09:12,840][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:09:13,341][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:09:13,845][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:09:14,348][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:09:14,844][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:09:15,351][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:09:15,847][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:09:16,345][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:09:16,842][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:09:17,337][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:09:17,850][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:09:18,352][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:09:18,852][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:09:19,351][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:09:19,855][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:09:20,356][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:09:20,858][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:09:21,366][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:09:21,870][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:09:22,373][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:09:22,873][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:09:23,376][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:09:23,878][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:09:24,382][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:09:24,885][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:09:25,387][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:09:25,887][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:09:26,392][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:09:26,895][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:09:27,396][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:09:27,900][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:09:28,415][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:09:28,918][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:09:29,429][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:09:29,933][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:09:30,432][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:09:30,949][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:09:31,452][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:09:33,142][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:09:33,703][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:09:34,220][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:09:34,718][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:09:35,219][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:09:35,720][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:09:36,219][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:09:36,714][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:09:37,212][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:09:37,708][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:09:38,205][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:09:38,709][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:09:39,207][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:09:39,706][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10201 tokens.
[2025-11-13 09:09:40,616][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:34
[2025-11-13 09:09:41,304][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:09:41,306][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:09:41,307][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:09:42,237][__main__][INFO] - Iteration 678 took 1m 4s (41.89% Gen, 56.66% Train). Generation: 27s, Training: 36s. Estimated remaining time: 42h 48m 18s. Estimated total time: 53h 48m 18s. Time estimates for 10 more iterations: 10m 45s, 100 more iterations: 1h 47m 36s, 500 more iterations: 8h 58m 3s.
[2025-11-13 09:09:42,241][__main__][INFO] - Starting iteration 678.
[2025-11-13 09:09:42,725][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:09:42,726][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:10:17,422][__main__][INFO] - Number of regex retries in iteration 678: 0
[2025-11-13 09:10:17,423][__main__][INFO] - agents played in iteration 678 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:10:18,249][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:18,287][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:18,314][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:18,339][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:18,339][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:10:18,340][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:10:19,136][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:10:19,588][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:10:20,091][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:10:20,595][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:10:21,100][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:10:21,602][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:10:22,103][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:10:22,606][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:10:23,113][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:10:23,613][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:10:24,114][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:10:24,614][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:10:25,120][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:10:25,619][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:10:26,120][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:10:26,621][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:10:27,123][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:10:27,626][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:10:28,126][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:10:28,630][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:10:29,145][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:10:29,646][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:10:30,159][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:10:30,667][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:10:31,168][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:10:31,675][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:10:32,179][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:10:32,683][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:10:33,187][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:10:33,691][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:10:34,195][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:10:34,698][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:10:35,202][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:10:35,707][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:10:36,208][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:10:36,710][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:10:37,215][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:10:37,716][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:10:38,219][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:10:38,725][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:10:39,230][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:10:39,731][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:10:40,234][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:10:40,754][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:10:41,257][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:10:41,758][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:10:42,273][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:10:42,777][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:10:43,282][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:10:43,782][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:10:44,284][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:10:44,788][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:10:45,287][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:10:45,790][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:10:46,292][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:10:46,799][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:10:47,303][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:10:47,806][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:10:48,305][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:10:48,809][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:10:49,307][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:10:49,805][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:10:50,306][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:10:50,807][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:10:51,307][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10139 tokens.
[2025-11-13 09:10:52,168][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.05%, ΔTime: 00:00:33
[2025-11-13 09:10:52,956][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:10:52,958][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:10:52,961][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:10:53,981][__main__][INFO] - Iteration 679 took 1m 11s (48.69% Gen, 49.87% Train). Generation: 34s, Training: 35s. Estimated remaining time: 48h 21m 36s. Estimated total time: 59h 22m 48s. Time estimates for 10 more iterations: 11m 52s, 100 more iterations: 1h 58m 45s, 500 more iterations: 9h 53m 48s.
[2025-11-13 09:10:53,983][__main__][INFO] - Starting iteration 679.
[2025-11-13 09:10:54,472][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:10:54,472][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:11:21,468][__main__][INFO] - Number of regex retries in iteration 679: 0
[2025-11-13 09:11:21,468][__main__][INFO] - agents played in iteration 679 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:11:22,230][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:22,255][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:22,280][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:22,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:22,304][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:11:22,305][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:11:23,105][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:11:23,559][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:11:24,077][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:11:24,571][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:11:25,067][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:11:25,568][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:11:26,063][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:11:26,568][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:11:27,067][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:11:27,568][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:11:28,072][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:11:28,593][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:11:29,094][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:11:29,592][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:11:30,090][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:11:30,591][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:11:31,091][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:11:31,592][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:11:32,095][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:11:32,595][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:11:33,099][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:11:33,599][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:11:34,100][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:11:34,605][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:11:35,105][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:11:35,606][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:11:36,108][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:11:36,609][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:11:37,113][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:11:37,614][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:11:38,112][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:11:38,618][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:11:39,118][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:11:39,623][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:11:40,128][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:11:40,632][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:11:41,143][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:11:41,645][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:11:42,147][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:11:42,655][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:11:43,160][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:11:43,679][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:11:44,185][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:11:44,689][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:11:45,190][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:11:45,696][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:11:46,200][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:11:46,706][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:11:47,210][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:11:47,718][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:11:48,219][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:11:48,724][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:11:49,227][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:11:49,730][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:11:50,232][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:11:50,733][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:11:51,234][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:11:51,736][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:11:52,235][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:11:52,738][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:11:53,240][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:11:53,746][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:11:54,269][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:11:54,769][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:11:55,270][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10141 tokens.
[2025-11-13 09:11:56,175][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.44%, ΔTime: 00:00:33
[2025-11-13 09:11:56,958][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:11:56,960][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:11:56,962][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:11:58,033][__main__][INFO] - Iteration 680 took 1m 3s (42.47% Gen, 55.84% Train). Generation: 26s, Training: 35s. Estimated remaining time: 41h 55m 49s. Estimated total time: 52h 58m 5s. Time estimates for 10 more iterations: 10m 35s, 100 more iterations: 1h 45m 56s, 500 more iterations: 8h 49m 40s.
[2025-11-13 09:11:58,035][__main__][INFO] - Starting iteration 680.
[2025-11-13 09:11:58,519][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 67 and human policies 1.
[2025-11-13 09:11:58,520][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:12:25,073][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:12:29,855][__main__][INFO] - Number of regex retries in iteration 680: 1
[2025-11-13 09:12:29,856][__main__][INFO] - agents played in iteration 680 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:12:30,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:30,719][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:30,743][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:30,765][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.42%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:30,766][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:12:30,766][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:12:31,596][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:12:32,054][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:12:32,559][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:12:33,056][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:12:33,551][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:12:34,060][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:12:34,558][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:12:35,080][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:12:35,577][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:12:36,075][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:12:36,577][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:12:37,083][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:12:37,590][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:12:38,091][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:12:38,591][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:12:39,098][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:12:39,596][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:12:40,099][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:12:40,605][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:12:41,106][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:12:41,616][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:12:42,116][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:12:42,618][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:12:43,129][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:12:43,629][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:12:44,130][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:12:44,631][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:12:45,134][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:12:45,638][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:12:46,139][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:12:46,639][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:12:47,141][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:12:47,651][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:12:48,155][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:12:48,670][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:12:49,171][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:12:49,674][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:12:50,174][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:12:50,689][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:12:51,192][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:12:51,698][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:12:52,195][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:12:52,697][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:12:53,203][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:12:53,704][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:12:54,205][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:12:54,709][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:12:55,210][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:12:55,711][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:12:56,234][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:12:56,743][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:12:57,253][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:12:57,759][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:12:58,263][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:12:58,772][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:12:59,278][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:12:59,789][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:13:00,292][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:13:00,792][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:13:01,303][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:13:01,803][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:13:02,305][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:13:02,808][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:13:03,308][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:13:03,812][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10112 tokens.
[2025-11-13 09:13:04,728][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:33
[2025-11-13 09:13:05,489][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:13:05,491][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:13:05,493][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:13:07,438][__main__][INFO] - Iteration 681 took 1m 8s (45.47% Gen, 51.71% Train). Generation: 31s, Training: 35s. Estimated remaining time: 46h 22m 34s. Estimated total time: 57h 26m 0s. Time estimates for 10 more iterations: 11m 29s, 100 more iterations: 1h 54m 52s, 500 more iterations: 9h 34m 20s.
[2025-11-13 09:13:07,440][__main__][INFO] - Starting iteration 681.
[2025-11-13 09:13:07,940][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:13:07,940][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:13:38,306][__main__][INFO] - Number of regex retries in iteration 681: 0
[2025-11-13 09:13:38,306][__main__][INFO] - agents played in iteration 681 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:13:39,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:39,193][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:39,218][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:39,241][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:39,241][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:13:39,243][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:13:40,052][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:13:40,510][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:13:41,015][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:13:41,514][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:13:42,011][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:13:42,509][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:13:43,013][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:13:43,515][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:13:44,011][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:13:44,516][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:13:45,011][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:13:45,506][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:13:46,002][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:13:46,498][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:13:47,005][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:13:47,503][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:13:48,001][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:13:48,512][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:13:49,012][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:13:49,524][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:13:50,026][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:13:50,527][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:13:51,033][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:13:51,534][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:13:52,031][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:13:52,534][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:13:53,044][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:13:53,549][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:13:54,051][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:13:54,555][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:13:55,067][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:13:55,572][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:13:56,077][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:13:56,584][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:13:57,086][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:13:57,598][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:13:58,103][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:13:58,606][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:13:59,108][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:13:59,611][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:14:00,115][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:14:00,614][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:14:01,117][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:14:01,623][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:14:02,125][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:14:02,627][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:14:03,131][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:14:03,631][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:14:04,131][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:14:04,635][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:14:05,137][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:14:05,639][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:14:06,145][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:14:06,647][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:14:07,154][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:14:07,654][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:14:08,155][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:14:08,658][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:14:09,156][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:14:09,656][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:14:10,162][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:14:10,662][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:14:11,174][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:14:11,672][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:14:12,175][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10077 tokens.
[2025-11-13 09:14:13,054][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:33
[2025-11-13 09:14:13,841][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:14:13,843][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:14:13,845][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:14:14,858][__main__][INFO] - Iteration 682 took 1m 6s (45.38% Gen, 53.11% Train). Generation: 30s, Training: 35s. Estimated remaining time: 44h 41m 22s. Estimated total time: 55h 45m 56s. Time estimates for 10 more iterations: 11m 9s, 100 more iterations: 1h 51m 31s, 500 more iterations: 9h 17m 39s.
[2025-11-13 09:14:14,860][__main__][INFO] - Starting iteration 682.
[2025-11-13 09:14:15,340][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:14:15,341][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:14:40,469][__main__][INFO] - Number of regex retries in iteration 682: 0
[2025-11-13 09:14:40,470][__main__][INFO] - agents played in iteration 682 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:14:41,392][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:41,418][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:41,444][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:41,467][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:41,468][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:14:41,469][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:14:42,284][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:14:42,739][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:14:43,245][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:14:43,748][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:14:44,250][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:14:44,748][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:14:45,248][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:14:45,747][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:14:46,245][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:14:46,743][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:14:47,241][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:14:47,741][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:14:48,239][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:14:48,736][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:14:49,233][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:14:49,728][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:14:50,232][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:14:50,729][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:14:51,226][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:14:51,728][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:14:52,225][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:14:52,724][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:14:53,221][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:14:53,722][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:14:54,227][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:14:54,728][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:14:55,232][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:14:55,735][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:14:56,236][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:14:56,739][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:14:57,245][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:14:59,000][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:14:59,527][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:15:00,031][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:15:00,531][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:15:01,039][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:15:01,538][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:15:02,044][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:15:02,545][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:15:03,051][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:15:03,555][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:15:04,057][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:15:04,558][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:15:05,061][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:15:05,567][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:15:06,074][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:15:06,574][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:15:07,074][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:15:07,586][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:15:08,092][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:15:08,601][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:15:09,099][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:15:09,599][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:15:10,100][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:15:10,602][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:15:11,102][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:15:11,603][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:15:12,104][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:15:12,619][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:15:13,124][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:15:13,629][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:15:14,132][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:15:14,634][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:15:15,139][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:15:15,638][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10125 tokens.
[2025-11-13 09:15:16,610][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.18%, ΔTime: 00:00:34
[2025-11-13 09:15:17,271][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:15:17,273][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:15:17,275][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:15:18,213][__main__][INFO] - Iteration 683 took 1m 2s (39.97% Gen, 58.54% Train). Generation: 25s, Training: 36s. Estimated remaining time: 41h 18m 4s. Estimated total time: 52h 23m 41s. Time estimates for 10 more iterations: 10m 28s, 100 more iterations: 1h 44m 47s, 500 more iterations: 8h 43m 56s.
[2025-11-13 09:15:18,215][__main__][INFO] - Starting iteration 683.
[2025-11-13 09:15:18,695][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:15:18,696][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:15:50,023][__main__][INFO] - Number of regex retries in iteration 683: 0
[2025-11-13 09:15:50,024][__main__][INFO] - agents played in iteration 683 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:15:50,868][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:50,893][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:50,918][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:50,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:50,941][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:15:50,942][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:15:51,752][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:15:52,210][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:15:52,716][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:15:53,225][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:15:53,722][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:15:54,222][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:15:54,722][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:15:55,220][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:15:55,725][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:15:56,220][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:15:56,716][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:15:57,230][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:15:57,725][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:15:58,220][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:15:58,720][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:15:59,215][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:15:59,721][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:16:00,221][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:16:00,721][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:16:01,217][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:16:01,718][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:16:02,214][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:16:02,713][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:16:03,209][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:16:03,721][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:16:04,215][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:16:04,713][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:16:05,214][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:16:05,712][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:16:06,222][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:16:06,721][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:16:07,221][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:16:07,727][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:16:08,227][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:16:08,727][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:16:09,229][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:16:09,732][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:16:10,233][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:16:10,736][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:16:11,236][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:16:11,741][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:16:12,242][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:16:12,742][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:16:13,240][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:16:13,741][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:16:14,247][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:16:14,748][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:16:15,250][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:16:15,751][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:16:16,250][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:16:16,752][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:16:17,256][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:16:17,756][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:16:18,256][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:16:18,756][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:16:19,257][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:16:19,760][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:16:20,260][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:16:20,760][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:16:21,260][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:16:21,760][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:16:22,259][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:16:22,762][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:16:23,268][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:16:23,766][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9986 tokens.
[2025-11-13 09:16:24,618][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.03%, ΔTime: 00:00:32
[2025-11-13 09:16:25,370][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:16:25,372][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:16:25,374][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:16:26,402][__main__][INFO] - Iteration 684 took 1m 7s (46.27% Gen, 52.21% Train). Generation: 31s, Training: 35s. Estimated remaining time: 45h 18m 36s. Estimated total time: 56h 25m 21s. Time estimates for 10 more iterations: 11m 17s, 100 more iterations: 1h 52m 50s, 500 more iterations: 9h 24m 13s.
[2025-11-13 09:16:26,404][__main__][INFO] - Starting iteration 684.
[2025-11-13 09:16:26,882][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:16:26,883][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:16:43,116][mllm.models.large_language_model_local][WARNING] - Response Proposal: x hats, y books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:16:46,696][mllm.models.large_language_model_local][WARNING] - Response Proposal: x hats, 10 books, 10 balls

In this round, I propose to keep all 10 balls and all 10 books, while keeping only a few hats. Given Alice's lower value for hats and my own value of 10 for balls, it makes sense to claim as many balls and books as possible to maximize my points. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:16:54,078][__main__][INFO] - Number of regex retries in iteration 684: 2
[2025-11-13 09:16:54,079][__main__][INFO] - agents played in iteration 684 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:16:54,986][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:55,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:55,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:55,071][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:55,072][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:16:55,073][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:16:55,875][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:16:56,326][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:16:56,831][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:16:57,331][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:16:57,829][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:16:58,329][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:16:58,833][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:16:59,342][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:16:59,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:17:00,338][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:17:00,837][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:17:01,333][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:17:01,838][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:17:02,333][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:17:02,829][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:17:03,335][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:17:03,831][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:17:04,329][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:17:04,826][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:17:05,321][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:17:05,828][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:17:06,324][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:17:06,819][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:17:07,316][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:17:07,811][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:17:08,308][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:17:08,804][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:17:09,304][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:17:09,815][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:17:10,316][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:17:10,843][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:17:11,344][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:17:11,848][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:17:12,354][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:17:12,854][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:17:13,358][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:17:13,860][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:17:14,367][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:17:14,869][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:17:15,373][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:17:15,874][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:17:16,377][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:17:16,877][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:17:17,377][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:17:17,882][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:17:18,382][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:17:18,882][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:17:19,382][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:17:20,734][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:17:21,379][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:17:21,886][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:17:22,388][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:17:22,893][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:17:23,396][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:17:23,900][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:17:24,411][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:17:24,913][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:17:25,416][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:17:25,920][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:17:26,427][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:17:26,940][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:17:27,441][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:17:27,944][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:17:28,458][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:17:28,959][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10010 tokens.
[2025-11-13 09:17:29,928][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 61.97%, ΔTime: 00:00:34
[2025-11-13 09:17:30,612][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:17:30,614][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:17:30,616][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:17:31,538][__main__][INFO] - Iteration 685 took 1m 4s (42.06% Gen, 56.51% Train). Generation: 27s, Training: 36s. Estimated remaining time: 42h 44m 58s. Estimated total time: 53h 52m 48s. Time estimates for 10 more iterations: 10m 46s, 100 more iterations: 1h 47m 45s, 500 more iterations: 8h 58m 48s.
[2025-11-13 09:17:31,541][__main__][INFO] - Starting iteration 685.
[2025-11-13 09:17:32,032][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:17:32,033][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:18:05,723][__main__][INFO] - Number of regex retries in iteration 685: 0
[2025-11-13 09:18:05,724][__main__][INFO] - agents played in iteration 685 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:18:06,565][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:06,592][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:06,619][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:06,642][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:06,643][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:18:06,643][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:18:07,445][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:18:07,911][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:18:08,413][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:18:08,911][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:18:09,410][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:18:09,908][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:18:10,407][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:18:10,910][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:18:11,408][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:18:11,912][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:18:12,411][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:18:12,921][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:18:13,423][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:18:13,926][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:18:14,438][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:18:14,944][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:18:15,454][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:18:15,955][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:18:16,453][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:18:16,963][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:18:17,467][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:18:17,972][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:18:18,474][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:18:18,978][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:18:19,479][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:18:19,983][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:18:20,490][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:18:20,992][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:18:21,497][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:18:22,000][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:18:22,503][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:18:23,004][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:18:23,513][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:18:24,017][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:18:24,518][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:18:25,023][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:18:25,523][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:18:26,043][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:18:26,544][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:18:27,045][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:18:27,547][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:18:28,049][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:18:28,553][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:18:29,055][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:18:29,556][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:18:30,067][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:18:30,570][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:18:31,071][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:18:31,572][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:18:32,073][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:18:32,578][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:18:33,080][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:18:33,581][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:18:34,088][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:18:34,589][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:18:35,091][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:18:35,593][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:18:36,090][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:18:36,592][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:18:37,089][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:18:37,588][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:18:38,089][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:18:38,587][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:18:39,086][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:18:39,586][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10055 tokens.
[2025-11-13 09:18:40,460][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.18%, Current % of VRAM taken: 60.44%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 09:18:41,201][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:18:41,203][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:18:41,204][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:18:42,263][__main__][INFO] - Iteration 686 took 1m 10s (47.97% Gen, 50.52% Train). Generation: 33s, Training: 35s. Estimated remaining time: 47h 22m 37s. Estimated total time: 58h 31m 37s. Time estimates for 10 more iterations: 11m 42s, 100 more iterations: 1h 57m 3s, 500 more iterations: 9h 45m 16s.
[2025-11-13 09:18:42,265][__main__][INFO] - Starting iteration 686.
[2025-11-13 09:18:42,747][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:18:42,747][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:18:59,231][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:18:59,312][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 10 books, 10 balls<tool_call>
 did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:19:09,107][__main__][INFO] - Number of regex retries in iteration 686: 2
[2025-11-13 09:19:09,108][__main__][INFO] - agents played in iteration 686 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:19:09,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:09,991][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:10,017][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:10,041][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:10,041][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:19:10,042][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:19:10,863][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:19:11,319][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:19:11,816][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:19:12,310][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:19:12,805][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:19:13,302][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:19:13,797][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:19:14,296][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:19:14,795][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:19:15,291][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:19:15,788][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:19:16,283][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:19:16,776][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:19:17,280][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:19:17,776][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:19:18,278][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:19:18,771][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:19:19,268][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:19:19,764][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:19:20,261][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:19:20,764][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:19:21,270][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:19:21,772][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:19:22,279][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:19:22,778][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:19:23,285][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:19:23,789][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:19:24,292][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:19:24,794][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:19:25,301][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:19:25,802][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:19:26,306][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:19:26,811][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:19:27,316][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:19:27,820][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:19:28,321][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:19:28,824][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:19:29,328][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:19:29,830][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:19:30,336][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:19:30,841][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:19:31,349][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:19:31,852][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:19:32,356][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:19:32,873][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:19:33,375][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:19:33,880][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:19:34,384][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:19:34,888][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:19:35,394][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:19:35,896][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:19:36,394][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:19:36,896][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:19:37,394][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:19:37,892][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:19:38,390][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:19:38,889][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:19:39,395][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:19:39,893][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:19:40,392][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:19:40,891][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:19:41,389][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:19:41,893][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:19:42,390][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:19:42,889][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10111 tokens.
[2025-11-13 09:19:43,781][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:32
[2025-11-13 09:19:44,551][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:19:44,552][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:19:44,554][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:19:45,611][__main__][INFO] - Iteration 687 took 1m 2s (41.93% Gen, 56.39% Train). Generation: 26s, Training: 35s. Estimated remaining time: 41h 13m 10s. Estimated total time: 52h 23m 14s. Time estimates for 10 more iterations: 10m 28s, 100 more iterations: 1h 44m 46s, 500 more iterations: 8h 43m 52s.
[2025-11-13 09:19:45,613][__main__][INFO] - Starting iteration 687.
[2025-11-13 09:19:46,105][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:19:46,106][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:20:17,491][__main__][INFO] - Number of regex retries in iteration 687: 0
[2025-11-13 09:20:17,492][__main__][INFO] - agents played in iteration 687 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:20:18,371][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:18,422][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:18,460][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:18,490][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:18,491][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:20:18,492][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:20:19,339][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:20:19,795][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:20:20,307][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:20:20,806][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:20:21,308][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:20:21,814][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:20:22,316][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:20:22,819][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:20:23,322][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:20:23,824][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:20:24,330][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:20:24,835][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:20:25,344][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:20:25,845][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:20:26,346][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:20:26,861][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:20:27,366][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:20:27,874][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:20:28,377][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:20:28,882][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:20:29,391][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:20:29,894][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:20:30,397][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:20:30,901][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:20:31,403][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:20:31,910][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:20:32,414][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:20:32,915][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:20:33,419][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:20:33,919][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:20:34,425][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:20:34,927][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:20:35,429][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:20:35,931][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:20:36,431][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:20:36,932][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:20:37,435][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:20:37,938][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:20:38,441][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:20:38,940][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:20:39,441][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:20:39,943][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:20:40,448][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:20:40,951][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:20:41,467][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:20:41,969][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:20:42,487][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:20:42,989][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:20:43,493][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:20:43,990][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:20:44,490][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:20:44,991][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:20:45,489][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:20:45,986][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:20:46,493][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:20:46,991][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:20:47,490][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:20:47,989][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:20:48,487][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:20:48,992][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:20:49,491][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:20:49,990][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:20:50,489][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:20:50,988][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:20:51,488][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10103 tokens.
[2025-11-13 09:20:52,371][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.06%, Current % of VRAM taken: 60.32%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:33
[2025-11-13 09:20:53,017][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:20:53,019][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:20:53,020][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:20:54,007][__main__][INFO] - Iteration 688 took 1m 7s (46.22% Gen, 52.32% Train). Generation: 31s, Training: 35s. Estimated remaining time: 45h 23m 54s. Estimated total time: 56h 35m 6s. Time estimates for 10 more iterations: 11m 19s, 100 more iterations: 1h 53m 10s, 500 more iterations: 9h 25m 51s.
[2025-11-13 09:20:54,009][__main__][INFO] - Starting iteration 688.
[2025-11-13 09:20:54,494][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:20:54,494][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:21:09,572][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:21:19,996][__main__][INFO] - Number of regex retries in iteration 688: 1
[2025-11-13 09:21:19,997][__main__][INFO] - agents played in iteration 688 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:21:20,985][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:21,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:21,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:21,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:21,062][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:21:21,063][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:21:21,872][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:21:22,326][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:21:22,829][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:21:23,329][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:21:23,825][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:21:24,324][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:21:24,826][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:21:25,323][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:21:25,820][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:21:26,315][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:21:26,812][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:21:27,309][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:21:27,809][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:21:28,313][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:21:28,812][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:21:29,319][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:21:29,822][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:21:30,318][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:21:30,821][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:21:31,321][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:21:31,822][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:21:32,327][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:21:32,831][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:21:33,335][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:21:33,837][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:21:34,349][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:21:34,850][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:21:35,351][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:21:35,858][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:21:36,364][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:21:36,867][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:21:37,378][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:21:37,883][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:21:38,385][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:21:38,898][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:21:39,402][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:21:39,904][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:21:40,412][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:21:40,917][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:21:41,420][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:21:41,928][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:21:42,427][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:21:42,929][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:21:43,431][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:21:43,933][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:21:44,436][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:21:44,938][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:21:45,438][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:21:45,943][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:21:46,466][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:21:47,000][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:21:47,506][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:21:48,015][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:21:48,518][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:21:49,021][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:21:49,523][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:21:50,028][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:21:50,530][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:21:51,035][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:21:51,534][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:21:52,032][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:21:52,533][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:21:53,032][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:21:53,532][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:21:54,027][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10030 tokens.
[2025-11-13 09:21:54,897][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 13.96%, Current % of VRAM taken: 60.22%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:33
[2025-11-13 09:21:55,633][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:21:55,635][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:21:55,636][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:21:56,678][__main__][INFO] - Iteration 689 took 1m 2s (41.01% Gen, 57.31% Train). Generation: 25s, Training: 35s. Estimated remaining time: 40h 37m 4s. Estimated total time: 51h 49m 19s. Time estimates for 10 more iterations: 10m 21s, 100 more iterations: 1h 43m 38s, 500 more iterations: 8h 38m 13s.
[2025-11-13 09:21:56,680][__main__][INFO] - Starting iteration 689.
[2025-11-13 09:21:57,169][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:21:57,170][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:22:25,611][__main__][INFO] - Number of regex retries in iteration 689: 0
[2025-11-13 09:22:25,613][__main__][INFO] - agents played in iteration 689 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:22:26,504][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:26,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:26,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:26,621][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:26,622][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:22:26,623][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:22:27,456][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:22:27,908][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:22:28,410][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:22:28,916][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:22:29,417][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:22:29,915][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:22:30,408][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:22:30,905][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:22:31,414][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:22:31,916][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:22:32,418][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:22:32,916][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:22:33,413][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:22:33,914][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:22:34,411][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:22:34,908][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:22:35,417][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:22:35,911][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:22:36,413][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:22:36,912][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:22:37,411][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:22:37,910][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:22:38,408][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:22:38,906][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:22:39,406][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:22:39,907][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:22:40,415][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:22:40,916][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:22:41,418][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:22:41,924][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:22:42,427][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:22:42,930][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:22:43,435][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:22:43,937][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:22:44,439][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:22:44,941][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:22:45,444][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:22:45,951][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:22:46,452][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:22:46,955][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:22:47,456][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:22:47,965][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:22:48,469][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:22:48,973][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:22:49,478][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:22:49,989][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:22:50,493][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:22:51,004][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:22:51,510][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:22:52,012][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:22:52,520][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:22:53,022][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:22:53,525][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:22:54,031][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:22:54,535][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:22:55,041][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:22:55,542][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:22:56,049][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:22:56,552][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:22:57,052][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:22:57,553][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:22:58,055][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:22:58,556][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:22:59,063][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:22:59,565][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10087 tokens.
[2025-11-13 09:23:00,525][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:33
[2025-11-13 09:23:01,165][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:23:01,168][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:23:01,170][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:23:02,103][__main__][INFO] - Iteration 690 took 1m 4s (43.80% Gen, 54.76% Train). Generation: 28s, Training: 35s. Estimated remaining time: 42h 53m 23s. Estimated total time: 54h 6m 44s. Time estimates for 10 more iterations: 10m 49s, 100 more iterations: 1h 48m 13s, 500 more iterations: 9h 1m 7s.
[2025-11-13 09:23:02,106][__main__][INFO] - Starting iteration 690.
[2025-11-13 09:23:02,597][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 68 and human policies 1.
[2025-11-13 09:23:02,598][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:23:31,820][__main__][INFO] - Number of regex retries in iteration 690: 0
[2025-11-13 09:23:31,821][__main__][INFO] - agents played in iteration 690 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:23:32,651][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:32,678][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:32,705][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:32,728][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:32,728][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:23:32,730][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:23:33,542][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:23:33,995][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:23:34,500][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:23:34,997][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:23:35,493][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:23:35,992][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:23:36,494][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:23:36,992][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:23:37,489][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:23:37,985][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:23:38,480][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:23:38,976][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:23:39,475][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:23:39,975][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:23:40,471][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:23:40,967][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:23:41,465][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:23:41,961][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:23:42,458][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:23:42,955][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:23:43,450][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:23:43,948][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:23:44,444][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:23:44,941][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:23:45,443][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:23:45,941][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:23:46,444][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:23:46,945][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:23:47,449][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:23:47,950][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:23:48,459][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:23:48,966][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:23:49,468][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:23:49,973][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:23:50,480][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:23:50,980][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:23:51,481][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:23:51,985][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:23:52,484][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:23:52,988][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:23:53,491][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:23:53,993][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:23:54,499][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:23:55,001][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:23:55,504][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:23:56,008][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:23:56,508][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:23:57,024][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:23:57,528][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:23:58,028][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:23:58,536][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:23:59,036][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:23:59,540][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:24:00,039][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:24:00,540][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:24:01,074][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:24:01,580][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:24:02,081][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:24:02,580][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:24:03,080][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:24:03,583][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:24:04,085][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:24:04,586][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:24:05,096][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:24:05,597][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10031 tokens.
[2025-11-13 09:24:06,540][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.03%, Current % of VRAM taken: 60.29%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:33
[2025-11-13 09:24:07,291][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:24:07,293][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:24:07,295][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:24:09,262][__main__][INFO] - Iteration 691 took 1m 6s (43.83% Gen, 53.21% Train). Generation: 29s, Training: 35s. Estimated remaining time: 44h 18m 49s. Estimated total time: 55h 33m 17s. Time estimates for 10 more iterations: 11m 6s, 100 more iterations: 1h 51m 6s, 500 more iterations: 9h 15m 32s.
[2025-11-13 09:24:09,264][__main__][INFO] - Starting iteration 691.
[2025-11-13 09:24:09,748][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:24:09,748][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:24:22,642][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:24:32,735][__main__][INFO] - Number of regex retries in iteration 691: 1
[2025-11-13 09:24:32,736][__main__][INFO] - agents played in iteration 691 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:24:33,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:33,522][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:33,546][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:33,569][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.30%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:33,570][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:24:33,571][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:24:34,380][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:24:34,834][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:24:35,336][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:24:35,833][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:24:36,347][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:24:36,846][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:24:37,344][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:24:37,843][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:24:38,338][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:24:38,843][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:24:39,342][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:24:39,842][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:24:40,341][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:24:40,838][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:24:41,339][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:24:41,838][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:24:42,335][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:24:42,849][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:24:43,347][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:24:43,849][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:24:44,350][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:24:44,846][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:24:45,341][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:24:45,842][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:24:46,337][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:24:46,846][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:24:47,343][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:24:47,843][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:24:48,340][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:24:48,836][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:24:49,339][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:24:49,834][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:24:50,331][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:24:50,844][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:24:51,339][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:24:51,840][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:24:52,344][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:24:52,842][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:24:53,348][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:24:53,850][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:24:54,347][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:24:54,855][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:24:55,354][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:24:55,855][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:24:56,354][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:24:56,857][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:24:57,365][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:24:57,870][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:24:58,490][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:24:58,997][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:24:59,531][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:25:00,050][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:25:00,552][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:25:01,054][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:25:01,562][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:25:02,069][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:25:02,575][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:25:03,076][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:25:03,577][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:25:04,077][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:25:04,581][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:25:05,082][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:25:05,585][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:25:06,089][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:25:06,594][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9982 tokens.
[2025-11-13 09:25:07,534][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.15%, Current % of VRAM taken: 60.41%, Block Peak % of device VRAM: 62.01%, ΔTime: 00:00:33
[2025-11-13 09:25:08,197][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:25:08,199][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:25:08,202][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:25:09,125][__main__][INFO] - Iteration 692 took 59s (38.71% Gen, 59.73% Train). Generation: 22s, Training: 35s. Estimated remaining time: 38h 13m 26s. Estimated total time: 49h 28m 54s. Time estimates for 10 more iterations: 9m 53s, 100 more iterations: 1h 38m 57s, 500 more iterations: 8h 14m 49s.
[2025-11-13 09:25:09,127][__main__][INFO] - Starting iteration 692.
[2025-11-13 09:25:09,625][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:25:09,626][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:25:43,112][__main__][INFO] - Number of regex retries in iteration 692: 0
[2025-11-13 09:25:43,113][__main__][INFO] - agents played in iteration 692 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:25:43,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:43,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:44,006][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:44,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:44,031][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:25:44,032][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:25:44,847][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:25:45,299][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:25:45,801][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:25:46,297][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:25:46,796][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:25:47,301][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:25:47,803][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:25:48,299][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:25:48,800][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:25:49,296][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:25:49,793][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:25:50,288][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:25:50,784][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:25:51,280][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:25:51,778][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:25:52,286][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:25:52,786][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:25:53,287][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:25:53,788][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:25:54,288][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:25:54,788][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:25:55,292][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:25:55,793][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:25:56,295][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:25:56,799][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:25:57,300][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:25:57,813][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:25:58,312][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:25:58,815][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:25:59,326][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:25:59,828][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:26:00,337][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:26:00,837][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:26:01,337][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:26:01,846][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:26:02,345][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:26:02,849][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:26:03,350][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:26:03,852][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:26:04,358][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:26:04,857][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:26:05,359][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:26:05,865][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:26:06,366][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:26:06,870][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:26:07,370][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:26:07,872][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:26:08,375][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:26:08,875][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:26:09,375][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:26:09,876][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:26:10,403][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:26:10,904][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:26:11,403][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:26:11,900][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:26:12,398][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:26:12,895][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:26:13,396][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:26:13,898][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:26:14,393][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:26:14,893][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:26:15,393][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:26:15,896][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:26:16,401][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:26:16,902][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10059 tokens.
[2025-11-13 09:26:17,766][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:32
[2025-11-13 09:26:18,512][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:26:18,514][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:26:18,516][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:26:19,530][__main__][INFO] - Iteration 693 took 1m 9s (47.90% Gen, 50.64% Train). Generation: 33s, Training: 35s. Estimated remaining time: 46h 58m 39s. Estimated total time: 58h 15m 17s. Time estimates for 10 more iterations: 11m 39s, 100 more iterations: 1h 56m 30s, 500 more iterations: 9h 42m 32s.
[2025-11-13 09:26:19,533][__main__][INFO] - Starting iteration 693.
[2025-11-13 09:26:20,018][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:26:20,018][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:26:44,323][__main__][INFO] - Number of regex retries in iteration 693: 0
[2025-11-13 09:26:44,324][__main__][INFO] - agents played in iteration 693 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:26:45,245][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:45,269][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:45,293][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:45,315][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:45,316][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:26:45,317][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:26:46,126][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:26:46,580][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:26:47,083][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:26:47,580][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:26:48,079][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:26:48,586][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:26:49,086][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:26:49,587][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:26:50,092][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:26:50,591][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:26:51,090][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:26:51,591][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:26:52,087][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:26:52,586][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:26:53,087][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:26:53,584][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:26:54,088][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:26:54,586][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:26:55,088][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:26:55,588][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:26:56,088][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:26:56,587][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:26:57,085][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:26:57,582][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:26:58,084][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:26:58,588][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:26:59,092][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:26:59,596][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:27:00,101][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:27:00,602][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:27:01,107][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:27:01,610][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:27:02,115][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:27:02,619][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:27:03,121][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:27:03,623][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:27:04,125][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:27:04,632][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:27:05,131][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:27:05,633][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:27:06,136][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:27:06,636][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:27:07,153][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:27:07,653][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:27:08,153][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:27:08,655][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:27:09,156][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:27:09,668][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:27:10,167][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:27:10,667][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:27:11,169][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:27:11,670][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:27:12,171][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:27:12,675][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:27:13,174][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:27:13,685][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:27:14,188][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:27:14,686][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:27:15,188][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:27:15,686][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:27:16,186][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:27:16,687][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:27:17,188][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:27:17,692][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:27:18,190][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10077 tokens.
[2025-11-13 09:27:19,106][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:32
[2025-11-13 09:27:19,863][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:27:19,865][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:27:19,866][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:27:20,888][__main__][INFO] - Iteration 694 took 1m 0s (39.93% Gen, 58.39% Train). Generation: 24s, Training: 35s. Estimated remaining time: 39h 25m 55s. Estimated total time: 50h 43m 34s. Time estimates for 10 more iterations: 10m 8s, 100 more iterations: 1h 41m 27s, 500 more iterations: 8h 27m 15s.
[2025-11-13 09:27:20,891][__main__][INFO] - Starting iteration 694.
[2025-11-13 09:27:21,386][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:27:21,386][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:27:49,009][__main__][INFO] - Number of regex retries in iteration 694: 0
[2025-11-13 09:27:49,011][__main__][INFO] - agents played in iteration 694 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:27:49,992][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:50,042][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:50,082][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:50,112][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:50,113][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:27:50,114][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:27:51,054][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:27:51,512][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:27:52,022][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:27:52,518][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:27:53,020][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:27:53,517][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:27:54,011][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:27:54,507][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:27:55,003][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:27:55,503][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:27:56,006][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:27:56,509][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:27:57,008][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:27:57,511][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:27:58,041][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:27:58,538][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:27:59,042][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:27:59,543][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:28:00,046][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:28:00,548][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:28:01,049][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:28:01,552][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:28:02,053][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:28:02,566][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:28:03,074][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:28:03,574][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:28:04,077][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:28:04,584][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:28:05,089][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:28:05,590][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:28:06,090][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:28:06,603][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:28:07,103][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:28:07,605][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:28:08,110][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:28:08,609][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:28:09,113][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:28:09,612][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:28:10,112][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:28:10,612][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:28:11,115][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:28:11,615][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:28:12,122][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:28:12,619][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:28:13,119][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:28:13,617][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:28:14,113][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:28:14,621][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:28:15,120][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:28:15,618][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:28:16,120][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:28:16,620][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:28:17,121][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:28:17,621][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:28:18,121][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:28:18,621][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:28:19,124][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:28:19,624][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:28:20,119][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:28:20,615][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:28:21,116][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:28:21,613][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:28:22,115][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:28:22,612][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:28:23,110][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 9956 tokens.
[2025-11-13 09:28:24,128][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.00%, Current % of VRAM taken: 60.26%, Block Peak % of device VRAM: 61.99%, ΔTime: 00:00:33
[2025-11-13 09:28:24,829][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:28:24,831][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:28:24,832][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:28:25,859][__main__][INFO] - Iteration 695 took 1m 4s (42.85% Gen, 55.56% Train). Generation: 27s, Training: 35s. Estimated remaining time: 42h 24m 58s. Estimated total time: 53h 43m 42s. Time estimates for 10 more iterations: 10m 44s, 100 more iterations: 1h 47m 27s, 500 more iterations: 8h 57m 17s.
[2025-11-13 09:28:25,861][__main__][INFO] - Starting iteration 695.
[2025-11-13 09:28:26,386][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:28:26,386][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:28:43,313][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:28:52,177][__main__][INFO] - Number of regex retries in iteration 695: 1
[2025-11-13 09:28:52,177][__main__][INFO] - agents played in iteration 695 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:28:53,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:53,124][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:53,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:53,174][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:53,174][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:28:53,175][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:28:54,046][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:28:54,502][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:28:55,010][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:28:55,513][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:28:56,010][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:28:56,509][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:28:57,004][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:28:57,501][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:28:58,013][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:28:58,515][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:28:59,013][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:28:59,509][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:29:00,004][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:29:00,504][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:29:01,005][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:29:01,501][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:29:01,999][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:29:02,494][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:29:02,991][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:29:03,493][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:29:03,991][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:29:04,493][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:29:04,991][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:29:05,491][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:29:06,009][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:29:06,508][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:29:07,009][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:29:07,512][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:29:08,020][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:29:08,522][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:29:09,021][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:29:09,521][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:29:10,027][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:29:10,525][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:29:11,028][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:29:11,534][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:29:12,035][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:29:12,537][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:29:13,041][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:29:13,541][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:29:14,041][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:29:14,545][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:29:15,054][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:29:15,561][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:29:16,067][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:29:16,571][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:29:17,097][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:29:17,601][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:29:18,106][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:29:18,608][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:29:19,115][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:29:19,614][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:29:20,119][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:29:20,615][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:29:21,114][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:29:21,616][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:29:22,118][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:29:22,622][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:29:23,125][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:29:23,626][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:29:24,126][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:29:24,624][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:29:25,122][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:29:25,623][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:29:26,121][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10011 tokens.
[2025-11-13 09:29:26,973][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.15%, ΔTime: 00:00:32
[2025-11-13 09:29:27,708][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:29:27,709][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:29:27,711][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:29:28,709][__main__][INFO] - Iteration 696 took 1m 2s (41.38% Gen, 57.01% Train). Generation: 25s, Training: 35s. Estimated remaining time: 40h 36m 23s. Estimated total time: 51h 56m 10s. Time estimates for 10 more iterations: 10m 23s, 100 more iterations: 1h 43m 52s, 500 more iterations: 8h 39m 21s.
[2025-11-13 09:29:28,711][__main__][INFO] - Starting iteration 696.
[2025-11-13 09:29:29,180][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:29:29,181][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:29:44,724][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 0 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:29:55,926][__main__][INFO] - Number of regex retries in iteration 696: 1
[2025-11-13 09:29:55,927][__main__][INFO] - agents played in iteration 696 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:29:56,761][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:56,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:56,844][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:56,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:56,872][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:29:56,873][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:29:57,710][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:29:58,168][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:29:58,672][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:29:59,172][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:29:59,672][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:30:00,172][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:30:00,672][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:30:01,183][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:30:01,685][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:30:02,196][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:30:02,696][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:30:03,194][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:30:03,697][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:30:04,194][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:30:04,695][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:30:05,195][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:30:05,692][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:30:06,204][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:30:06,700][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:30:07,195][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:30:07,695][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:30:08,190][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:30:08,705][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:30:09,209][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:30:09,707][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:30:10,209][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:30:10,709][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:30:11,214][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:30:11,714][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:30:12,215][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:30:12,719][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:30:13,221][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:30:13,723][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:30:14,227][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:30:14,729][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:30:15,242][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:30:15,747][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:30:16,248][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:30:16,755][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:30:17,257][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:30:17,762][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:30:18,269][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:30:18,772][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:30:19,273][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:30:19,774][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:30:20,274][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:30:20,788][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:30:21,289][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:30:21,812][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:30:22,314][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:30:22,820][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:30:23,323][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:30:23,827][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:30:24,334][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:30:24,836][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:30:25,335][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:30:25,837][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:30:26,340][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:30:26,838][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:30:27,338][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:30:27,841][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:30:28,342][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:30:28,841][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:30:29,349][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:30:29,848][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10146 tokens.
[2025-11-13 09:30:30,780][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 09:30:31,440][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:30:31,443][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:30:31,444][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:30:32,395][__main__][INFO] - Iteration 697 took 1m 3s (42.31% Gen, 56.18% Train). Generation: 26s, Training: 35s. Estimated remaining time: 41h 19m 57s. Estimated total time: 52h 40m 48s. Time estimates for 10 more iterations: 10m 32s, 100 more iterations: 1h 45m 21s, 500 more iterations: 8h 46m 48s.
[2025-11-13 09:30:32,397][__main__][INFO] - Starting iteration 697.
[2025-11-13 09:30:32,879][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:30:32,880][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:31:02,487][__main__][INFO] - Number of regex retries in iteration 697: 0
[2025-11-13 09:31:02,488][__main__][INFO] - agents played in iteration 697 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:31:03,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:03,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:03,358][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:03,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:03,381][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:31:03,382][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:31:04,214][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:31:04,670][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:31:05,181][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:31:05,686][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:31:06,187][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:31:06,703][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:31:07,205][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:31:07,707][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:31:08,211][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:31:08,709][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:31:09,212][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:31:09,710][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:31:10,210][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:31:10,710][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:31:11,208][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:31:11,724][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:31:12,226][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:31:12,725][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:31:13,225][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:31:13,723][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:31:14,230][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:31:14,732][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:31:15,232][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:31:15,740][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:31:16,244][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:31:16,746][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:31:17,251][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:31:17,754][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:31:18,254][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:31:18,754][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:31:19,253][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:31:19,755][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:31:20,256][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:31:20,762][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:31:21,263][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:31:21,764][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:31:22,268][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:31:22,767][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:31:23,268][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:31:23,768][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:31:24,269][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:31:24,770][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:31:25,270][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:31:25,770][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:31:26,277][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:31:26,777][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:31:27,280][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:31:27,780][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:31:28,277][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:31:28,780][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:31:29,281][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:31:29,783][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:31:30,286][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:31:30,787][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:31:31,292][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:31:31,793][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:31:32,291][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:31:32,803][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:31:33,303][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:31:33,800][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:31:34,298][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:31:34,798][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:31:35,297][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:31:35,793][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:31:36,290][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10101 tokens.
[2025-11-13 09:31:37,150][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 09:31:37,909][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:31:37,911][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:31:37,912][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:31:38,920][__main__][INFO] - Iteration 698 took 1m 6s (44.83% Gen, 53.64% Train). Generation: 29s, Training: 35s. Estimated remaining time: 43h 40m 9s. Estimated total time: 55h 2m 6s. Time estimates for 10 more iterations: 11m 0s, 100 more iterations: 1h 50m 4s, 500 more iterations: 9h 10m 21s.
[2025-11-13 09:31:38,922][__main__][INFO] - Starting iteration 698.
[2025-11-13 09:31:39,409][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:31:39,410][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:32:06,784][__main__][INFO] - Number of regex retries in iteration 698: 0
[2025-11-13 09:32:06,785][__main__][INFO] - agents played in iteration 698 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:32:07,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:07,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:07,649][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:07,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:07,673][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:32:07,674][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:32:08,523][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:32:08,978][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:32:09,485][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:32:09,983][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:32:10,482][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:32:10,979][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:32:11,487][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:32:11,989][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:32:12,489][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:32:12,990][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:32:13,496][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:32:13,994][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:32:14,493][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:32:14,990][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:32:15,492][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:32:15,993][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:32:16,494][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:32:16,996][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:32:17,503][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:32:18,005][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:32:18,509][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:32:19,007][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:32:19,508][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:32:20,009][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:32:20,509][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:32:21,010][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:32:21,514][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:32:22,014][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:32:22,520][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:32:23,020][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:32:23,522][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:32:24,027][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:32:24,527][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:32:25,028][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:32:25,530][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:32:26,029][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:32:26,550][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:32:27,052][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:32:27,550][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:32:28,052][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:32:28,552][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:32:29,063][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:32:29,565][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:32:30,067][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:32:30,567][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:32:31,069][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:32:31,567][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:32:32,066][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:32:32,569][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:32:33,076][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:32:33,572][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:32:34,070][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:32:34,569][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:32:35,068][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:32:35,566][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:32:36,064][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:32:36,565][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:32:37,068][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:32:37,566][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:32:38,065][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:32:38,566][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:32:39,064][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:32:39,562][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:32:40,059][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:32:40,557][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10027 tokens.
[2025-11-13 09:32:41,483][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.10%, ΔTime: 00:00:32
[2025-11-13 09:32:42,235][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:32:42,237][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:32:42,239][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:32:44,034][__main__][INFO] - Iteration 699 took 1m 4s (42.36% Gen, 54.86% Train). Generation: 27s, Training: 35s. Estimated remaining time: 42h 28m 13s. Estimated total time: 53h 51m 15s. Time estimates for 10 more iterations: 10m 46s, 100 more iterations: 1h 47m 42s, 500 more iterations: 8h 58m 32s.
[2025-11-13 09:32:44,041][__main__][INFO] - Starting iteration 699.
[2025-11-13 09:32:44,528][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:32:44,529][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:33:22,629][__main__][INFO] - Number of regex retries in iteration 699: 0
[2025-11-13 09:33:22,630][__main__][INFO] - agents played in iteration 699 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:33:23,511][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:23,558][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:23,594][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:23,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:23,623][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:33:23,624][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:33:24,545][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:33:25,007][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:33:25,510][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:33:26,009][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:33:26,509][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:33:27,013][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:33:27,512][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:33:28,014][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:33:28,519][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:33:29,022][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:33:29,524][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:33:30,024][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:33:30,525][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:33:31,032][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:33:31,531][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:33:32,032][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:33:32,535][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:33:33,039][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:33:33,547][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:33:34,051][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:33:34,553][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:33:35,067][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:33:35,569][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:33:36,080][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:33:36,577][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:33:37,073][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:33:37,586][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:33:38,084][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:33:38,585][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:33:39,082][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:33:39,578][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:33:40,080][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:33:40,575][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:33:41,070][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:33:41,569][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:33:42,070][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:33:42,577][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:33:43,073][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:33:43,569][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:33:44,085][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:33:44,582][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:33:45,079][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:33:45,577][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:33:46,079][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:33:46,591][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:33:47,087][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:33:47,589][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:33:48,096][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:33:48,596][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:33:49,096][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:33:49,593][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:33:50,087][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:33:50,590][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:33:51,086][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:33:51,582][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:33:52,081][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:33:52,577][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:33:53,074][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:33:53,577][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:33:54,073][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:33:54,572][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:33:55,067][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:33:55,566][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:33:56,063][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:33:56,565][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10059 tokens.
[2025-11-13 09:33:57,404][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.14%, ΔTime: 00:00:32
[2025-11-13 09:33:58,138][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:33:58,140][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:33:58,141][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:33:59,173][__main__][INFO] - Iteration 700 took 1m 14s (51.04% Gen, 47.57% Train). Generation: 38s, Training: 35s. Estimated remaining time: 50h 48m 0s. Estimated total time: 62h 12m 18s. Time estimates for 10 more iterations: 12m 26s, 100 more iterations: 2h 4m 24s, 500 more iterations: 10h 22m 3s.
[2025-11-13 09:33:59,175][__main__][INFO] - Starting iteration 700.
[2025-11-13 09:33:59,661][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 69 and human policies 1.
[2025-11-13 09:33:59,662][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:34:25,075][__main__][INFO] - Number of regex retries in iteration 700: 0
[2025-11-13 09:34:25,076][__main__][INFO] - agents played in iteration 700 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:34:26,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:26,107][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:26,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:26,154][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:26,155][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:34:26,156][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:34:27,071][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:34:27,529][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:34:28,033][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:34:28,535][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:34:29,042][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:34:29,545][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:34:30,047][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:34:30,549][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:34:31,050][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:34:31,551][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:34:32,051][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:34:32,550][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:34:33,053][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:34:33,552][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:34:34,054][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:34:34,553][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:34:35,052][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:34:35,554][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:34:36,054][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:34:36,553][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:34:37,052][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:34:37,550][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:34:38,049][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:34:38,548][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:34:39,050][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:34:39,554][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:34:40,057][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:34:40,556][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:34:41,054][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:34:41,551][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:34:42,051][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:34:42,548][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:34:43,047][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:34:43,546][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:34:44,046][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:34:44,546][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:34:45,046][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:34:45,545][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:34:46,044][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:34:46,543][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:34:47,043][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:34:47,546][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:34:48,050][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:34:48,550][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:34:49,054][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:34:49,555][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:34:50,057][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:34:50,554][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:34:51,056][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:34:51,552][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:34:52,048][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:34:52,546][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:34:53,041][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:34:53,564][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:34:54,059][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:34:54,556][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:34:55,053][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:34:55,551][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:34:56,051][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:34:56,552][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:34:57,048][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:34:57,546][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:34:58,042][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:34:58,542][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:34:59,040][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10003 tokens.
[2025-11-13 09:34:59,874][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.06%, ΔTime: 00:00:32
[2025-11-13 09:35:00,656][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:35:00,658][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:35:00,660][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:35:02,880][__main__][INFO] - Iteration 701 took 1m 3s (40.20% Gen, 56.29% Train). Generation: 25s, Training: 35s. Estimated remaining time: 41h 15m 36s. Estimated total time: 52h 40m 57s. Time estimates for 10 more iterations: 10m 32s, 100 more iterations: 1h 45m 21s, 500 more iterations: 8h 46m 49s.
[2025-11-13 09:35:02,882][__main__][INFO] - Starting iteration 701.
[2025-11-13 09:35:03,369][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:35:03,370][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:35:24,039][__main__][INFO] - Number of regex retries in iteration 701: 0
[2025-11-13 09:35:24,040][__main__][INFO] - agents played in iteration 701 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:35:25,011][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:25,033][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:25,057][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:25,080][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:25,080][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:35:25,081][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:35:26,048][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:35:26,506][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:35:27,010][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:35:27,525][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:35:28,025][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:35:28,532][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:35:29,035][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:35:29,533][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:35:30,034][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:35:30,534][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:35:31,035][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:35:31,539][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:35:32,039][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:35:32,548][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:35:33,048][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:35:33,550][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:35:34,053][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:35:34,558][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:35:35,060][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:35:35,563][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:35:36,065][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:35:36,569][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:35:37,074][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:35:37,577][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:35:38,082][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:35:38,584][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:35:39,084][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:35:39,596][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:35:40,103][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:35:40,615][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:35:41,116][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:35:41,617][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:35:42,121][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:35:42,621][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:35:43,126][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:35:43,626][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:35:44,129][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:35:44,637][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:35:45,141][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:35:45,641][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:35:46,144][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:35:46,645][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:35:47,171][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:35:47,674][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:35:48,179][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:35:48,699][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:35:49,202][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:35:49,702][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:35:50,202][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:35:50,700][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:35:51,211][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:35:51,710][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:35:52,211][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:35:52,713][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:35:53,211][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:35:53,720][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:35:54,218][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:35:54,718][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:35:55,231][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:35:55,730][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:35:56,231][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:35:56,728][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:35:57,228][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:35:57,734][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:35:58,236][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10072 tokens.
[2025-11-13 09:35:59,172][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.20%, Current % of VRAM taken: 60.46%, Block Peak % of device VRAM: 62.29%, ΔTime: 00:00:33
[2025-11-13 09:35:59,911][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:35:59,912][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:35:59,914][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:36:00,928][__main__][INFO] - Iteration 702 took 57s (35.91% Gen, 62.33% Train). Generation: 20s, Training: 35s. Estimated remaining time: 36h 31m 37s. Estimated total time: 47h 57m 57s. Time estimates for 10 more iterations: 9m 35s, 100 more iterations: 1h 35m 55s, 500 more iterations: 7h 59m 39s.
[2025-11-13 09:36:00,930][__main__][INFO] - Starting iteration 702.
[2025-11-13 09:36:01,400][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:36:01,401][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:36:24,856][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:36:26,887][mllm.models.large_language_model_local][WARNING] - Response Proposal: 0 hats, 0 books, 0 balls

Given Bob's high value for balls and hats and our low values for these items, it's best to focus on books where we both have a high value. did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:36:27,791][__main__][INFO] - Number of regex retries in iteration 702: 2
[2025-11-13 09:36:27,792][__main__][INFO] - agents played in iteration 702 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:36:28,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:28,627][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:28,653][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:28,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:28,676][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:36:28,677][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:36:29,619][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:36:30,077][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:36:30,586][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:36:31,089][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:36:31,595][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:36:32,098][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:36:32,603][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:36:33,108][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:36:33,621][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:36:34,124][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:36:34,639][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:36:35,141][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:36:35,651][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:36:36,159][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:36:36,660][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:36:37,164][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:36:37,667][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:36:38,168][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:36:38,673][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:36:39,173][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:36:39,678][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:36:40,183][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:36:40,689][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:36:41,191][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:36:41,693][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:36:42,193][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:36:42,696][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:36:43,195][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:36:43,695][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:36:44,203][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:36:44,706][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:36:45,210][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:36:45,713][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:36:46,212][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:36:46,719][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:36:47,220][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:36:47,731][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:36:48,231][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:36:48,729][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:36:49,243][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:36:49,740][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:36:50,237][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:36:50,734][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:36:51,230][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:36:51,739][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:36:52,237][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:36:52,733][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:36:53,237][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:36:53,736][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:36:54,240][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:36:54,743][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:36:55,242][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:36:55,747][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:36:56,244][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:36:56,740][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:36:57,236][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:36:57,738][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:36:58,238][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:36:58,735][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:36:59,234][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:36:59,753][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:37:00,256][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:37:00,757][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:37:01,263][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:37:01,761][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10093 tokens.
[2025-11-13 09:37:02,604][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:32
[2025-11-13 09:37:03,329][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:37:03,330][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:37:03,332][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:37:04,340][__main__][INFO] - Iteration 703 took 1m 2s (41.93% Gen, 56.47% Train). Generation: 26s, Training: 35s. Estimated remaining time: 40h 59m 38s. Estimated total time: 52h 27m 0s. Time estimates for 10 more iterations: 10m 29s, 100 more iterations: 1h 44m 54s, 500 more iterations: 8h 44m 30s.
[2025-11-13 09:37:04,342][__main__][INFO] - Starting iteration 703.
[2025-11-13 09:37:04,856][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:37:04,856][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:37:28,648][__main__][INFO] - Number of regex retries in iteration 703: 0
[2025-11-13 09:37:28,648][__main__][INFO] - agents played in iteration 703 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:37:29,580][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:29,606][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:29,632][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:29,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:29,655][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:37:29,656][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:37:30,520][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:37:30,981][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:37:31,484][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:37:31,984][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:37:32,485][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:37:32,986][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:37:33,485][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:37:33,985][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:37:34,489][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:37:34,992][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:37:35,493][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:37:35,993][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:37:36,495][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:37:36,996][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:37:37,500][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:37:38,003][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:37:38,505][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:37:39,008][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:37:39,510][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:37:40,068][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:37:41,372][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:37:41,887][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:37:42,395][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:37:42,911][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:37:43,428][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:37:43,931][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:37:44,440][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:37:44,944][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:37:45,446][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:37:45,951][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:37:46,451][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:37:46,954][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:37:47,454][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:37:47,954][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:37:48,463][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:37:48,968][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:37:49,469][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:37:49,971][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:37:50,473][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:37:50,979][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:37:51,481][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:37:51,982][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:37:52,485][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:37:52,986][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:37:53,491][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:37:53,995][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:37:54,494][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:37:54,992][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:37:55,491][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:37:55,993][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:37:56,492][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:37:56,995][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:37:57,493][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:37:58,016][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:37:58,515][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:37:59,021][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:37:59,520][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:38:00,018][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:38:00,516][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:38:01,017][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:38:01,515][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:38:02,015][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:38:02,513][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:38:03,011][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:38:03,510][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10063 tokens.
[2025-11-13 09:38:04,446][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.20%, ΔTime: 00:00:33
[2025-11-13 09:38:05,093][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:38:05,095][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:38:05,096][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:38:06,114][__main__][INFO] - Iteration 704 took 1m 1s (38.84% Gen, 59.50% Train). Generation: 23s, Training: 36s. Estimated remaining time: 39h 34m 33s. Estimated total time: 51h 2m 58s. Time estimates for 10 more iterations: 10m 12s, 100 more iterations: 1h 42m 5s, 500 more iterations: 8h 30m 29s.
[2025-11-13 09:38:06,117][__main__][INFO] - Starting iteration 704.
[2025-11-13 09:38:06,608][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:38:06,609][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:38:39,459][__main__][INFO] - Number of regex retries in iteration 704: 0
[2025-11-13 09:38:39,460][__main__][INFO] - agents played in iteration 704 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:38:40,457][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:40,484][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:40,511][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:40,535][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:40,536][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:38:40,537][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:38:41,466][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:38:41,922][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:38:42,432][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:38:42,935][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:38:43,441][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:38:43,954][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:38:44,455][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:38:44,956][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:38:45,463][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:38:45,966][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:38:46,470][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:38:46,972][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:38:47,483][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:38:47,984][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:38:48,484][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:38:48,993][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:38:49,493][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:38:49,994][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:38:50,495][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:38:50,995][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:38:51,498][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:38:51,998][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:38:52,499][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:38:53,008][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:38:53,510][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:38:54,018][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:38:54,525][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:38:55,030][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:38:55,535][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:38:56,043][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:38:56,548][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:38:57,065][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:38:57,570][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:38:58,071][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:38:58,575][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:38:59,078][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:38:59,582][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:39:00,085][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:39:00,585][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:39:01,088][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:39:01,589][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:39:02,106][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:39:02,608][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:39:03,111][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:39:03,618][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:39:04,117][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:39:04,618][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:39:05,160][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:39:05,662][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:39:06,163][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:39:06,664][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:39:07,164][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:39:07,664][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:39:08,162][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:39:08,660][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:39:09,162][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:39:09,663][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:39:10,173][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:39:10,669][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:39:11,165][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:39:11,670][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:39:12,170][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:39:12,675][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:39:13,174][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:39:13,677][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10015 tokens.
[2025-11-13 09:39:14,535][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.16%, Current % of VRAM taken: 60.42%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:33
[2025-11-13 09:39:15,296][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:39:15,298][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:39:15,301][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:39:16,600][__main__][INFO] - Iteration 705 took 1m 9s (46.93% Gen, 51.21% Train). Generation: 32s, Training: 35s. Estimated remaining time: 46h 50m 3s. Estimated total time: 58h 19m 38s. Time estimates for 10 more iterations: 11m 39s, 100 more iterations: 1h 56m 39s, 500 more iterations: 9h 43m 16s.
[2025-11-13 09:39:16,602][__main__][INFO] - Starting iteration 705.
[2025-11-13 09:39:17,083][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:39:17,083][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:39:42,065][__main__][INFO] - Number of regex retries in iteration 705: 0
[2025-11-13 09:39:42,065][__main__][INFO] - agents played in iteration 705 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:39:42,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:42,990][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:43,016][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:43,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:43,039][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:39:43,040][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:39:43,847][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:39:44,300][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:39:44,801][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:39:45,297][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:39:45,793][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:39:46,289][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:39:46,791][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:39:47,293][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:39:47,793][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:39:48,313][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:39:48,818][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:39:49,313][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:39:49,813][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:39:50,312][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:39:50,820][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:39:51,320][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:39:51,821][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:39:52,323][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:39:52,829][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:39:53,331][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:39:53,836][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:39:54,336][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:39:54,842][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:39:55,343][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:39:55,843][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:39:56,345][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:39:56,848][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:39:57,348][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:39:57,848][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:39:58,350][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:39:59,879][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:40:00,400][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:40:00,905][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:40:01,419][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:40:01,923][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:40:02,426][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:40:02,935][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:40:03,439][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:40:03,946][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:40:04,450][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:40:04,975][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:40:05,485][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:40:05,986][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:40:06,499][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:40:06,999][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:40:07,499][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:40:08,000][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:40:08,507][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:40:09,005][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:40:09,510][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:40:10,010][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:40:10,514][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:40:11,015][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:40:11,518][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:40:12,017][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:40:12,517][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:40:13,027][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:40:13,532][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:40:14,033][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:40:14,535][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:40:15,036][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:40:15,536][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:40:16,038][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:40:16,542][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:40:17,045][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10024 tokens.
[2025-11-13 09:40:17,919][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:34
[2025-11-13 09:40:18,573][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:40:18,575][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:40:18,577][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:40:19,546][__main__][INFO] - Iteration 706 took 1m 2s (39.99% Gen, 58.45% Train). Generation: 24s, Training: 36s. Estimated remaining time: 40h 32m 32s. Estimated total time: 52h 3m 10s. Time estimates for 10 more iterations: 10m 24s, 100 more iterations: 1h 44m 6s, 500 more iterations: 8h 40m 31s.
[2025-11-13 09:40:19,548][__main__][INFO] - Starting iteration 706.
[2025-11-13 09:40:20,035][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:40:20,036][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:40:50,345][__main__][INFO] - Number of regex retries in iteration 706: 0
[2025-11-13 09:40:50,346][__main__][INFO] - agents played in iteration 706 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:40:51,336][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:51,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:51,390][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:51,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:51,414][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:40:51,415][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:40:52,301][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:40:52,761][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:40:53,265][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:40:53,770][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:40:54,274][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:40:54,786][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:40:55,292][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:40:55,792][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:40:56,312][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:40:56,814][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:40:57,315][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:40:57,817][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:40:58,324][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:40:58,832][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:40:59,333][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:40:59,840][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:41:00,345][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:41:00,847][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:41:01,348][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:41:01,851][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:41:02,353][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:41:02,861][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:41:03,363][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:41:03,866][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:41:04,371][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:41:04,873][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:41:05,374][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:41:05,878][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:41:06,380][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:41:06,885][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:41:07,385][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:41:07,888][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:41:08,394][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:41:08,898][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:41:09,414][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:41:09,913][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:41:10,416][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:41:10,918][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:41:11,418][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:41:11,928][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:41:12,426][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:41:12,927][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:41:13,434][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:41:13,937][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:41:14,437][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:41:14,939][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:41:15,441][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:41:15,947][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:41:16,449][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:41:16,951][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:41:17,454][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:41:17,956][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:41:18,459][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:41:18,961][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:41:19,463][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:41:19,967][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:41:20,473][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:41:20,974][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:41:21,483][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:41:21,983][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:41:22,497][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:41:23,000][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:41:23,502][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:41:24,003][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:41:24,503][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10101 tokens.
[2025-11-13 09:41:25,434][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.17%, ΔTime: 00:00:33
[2025-11-13 09:41:26,175][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:41:26,176][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:41:26,178][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:41:27,231][__main__][INFO] - Iteration 707 took 1m 7s (45.11% Gen, 53.32% Train). Generation: 30s, Training: 35s. Estimated remaining time: 44h 28m 3s. Estimated total time: 55h 59m 49s. Time estimates for 10 more iterations: 11m 11s, 100 more iterations: 1h 51m 59s, 500 more iterations: 9h 19m 58s.
[2025-11-13 09:41:27,233][__main__][INFO] - Starting iteration 707.
[2025-11-13 09:41:27,721][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:41:27,722][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:41:53,713][__main__][INFO] - Number of regex retries in iteration 707: 0
[2025-11-13 09:41:53,714][__main__][INFO] - agents played in iteration 707 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:41:54,493][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:54,521][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:54,547][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:54,569][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:54,570][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:41:54,571][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:41:55,390][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:41:55,978][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:41:56,482][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:41:56,981][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:41:57,480][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:41:57,982][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:41:58,478][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:41:58,983][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:41:59,488][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:41:59,991][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:42:00,494][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:42:00,996][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:42:01,498][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:42:02,001][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:42:02,505][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:42:03,011][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:42:03,528][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:42:04,030][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:42:04,534][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:42:05,034][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:42:05,534][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:42:06,038][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:42:06,538][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:42:07,039][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:42:07,544][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:42:08,044][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:42:08,548][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:42:09,049][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:42:09,552][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:42:10,061][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:42:10,563][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:42:11,065][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:42:11,565][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:42:12,067][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:42:12,571][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:42:13,078][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:42:13,582][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:42:14,086][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:42:14,588][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:42:15,100][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:42:15,601][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:42:16,103][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:42:16,620][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:42:17,126][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:42:17,629][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:42:18,136][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:42:18,637][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:42:19,145][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:42:19,651][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:42:20,155][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:42:20,656][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:42:21,156][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:42:21,660][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:42:22,168][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:42:22,667][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:42:23,168][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:42:23,669][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:42:24,171][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:42:24,669][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:42:25,169][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:42:25,670][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:42:26,174][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:42:26,673][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:42:27,173][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:42:27,675][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10183 tokens.
[2025-11-13 09:42:28,575][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:33
[2025-11-13 09:42:29,322][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:42:29,323][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:42:29,326][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:42:30,340][__main__][INFO] - Iteration 708 took 1m 2s (41.51% Gen, 56.87% Train). Generation: 25s, Training: 35s. Estimated remaining time: 40h 38m 11s. Estimated total time: 52h 10m 59s. Time estimates for 10 more iterations: 10m 26s, 100 more iterations: 1h 44m 21s, 500 more iterations: 8h 41m 49s.
[2025-11-13 09:42:30,343][__main__][INFO] - Starting iteration 708.
[2025-11-13 09:42:30,834][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:42:30,834][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:43:03,556][__main__][INFO] - Number of regex retries in iteration 708: 0
[2025-11-13 09:43:03,558][__main__][INFO] - agents played in iteration 708 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:43:04,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:04,420][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:04,447][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:04,470][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:04,471][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:43:04,472][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:43:05,296][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:43:05,758][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:43:06,266][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:43:06,772][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:43:07,276][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:43:07,782][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:43:08,286][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:43:08,787][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:43:09,294][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:43:09,796][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:43:10,298][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:43:10,803][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:43:11,307][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:43:11,813][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:43:12,318][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:43:12,822][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:43:13,330][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:43:13,831][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:43:14,343][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:43:14,847][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:43:15,349][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:43:15,861][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:43:16,370][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:43:16,872][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:43:17,378][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:43:17,882][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:43:18,391][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:43:18,894][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:43:19,393][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:43:19,900][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:43:20,406][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:43:20,907][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:43:21,421][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:43:21,925][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:43:22,444][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:43:22,952][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:43:23,458][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:43:23,964][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:43:24,470][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:43:24,972][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:43:25,475][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:43:25,975][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:43:26,478][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:43:26,983][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:43:27,484][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:43:27,984][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:43:28,489][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:43:28,994][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:43:29,495][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:43:29,998][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:43:30,501][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:43:31,003][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:43:31,507][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:43:32,010][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:43:32,511][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:43:33,015][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:43:33,522][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:43:34,023][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:43:34,527][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:43:35,028][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:43:35,525][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:43:36,028][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:43:36,524][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:43:37,023][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:43:37,523][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10140 tokens.
[2025-11-13 09:43:38,401][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.19%, Current % of VRAM taken: 60.45%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:33
[2025-11-13 09:43:39,103][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:43:39,105][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:43:39,106][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:43:40,052][__main__][INFO] - Iteration 709 took 1m 9s (47.28% Gen, 51.36% Train). Generation: 32s, Training: 35s. Estimated remaining time: 46h 6m 58s. Estimated total time: 57h 40m 56s. Time estimates for 10 more iterations: 11m 32s, 100 more iterations: 1h 55m 21s, 500 more iterations: 9h 36m 49s.
[2025-11-13 09:43:40,054][__main__][INFO] - Starting iteration 709.
[2025-11-13 09:43:40,538][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:43:40,539][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:44:01,082][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:44:12,045][__main__][INFO] - Number of regex retries in iteration 709: 1
[2025-11-13 09:44:12,046][__main__][INFO] - agents played in iteration 709 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:44:12,826][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:12,853][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:12,880][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:12,903][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:12,903][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:44:12,904][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:44:13,726][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:44:14,179][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:44:14,681][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:44:15,182][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:44:15,678][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:44:16,174][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:44:16,671][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:44:17,178][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:44:17,683][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:44:18,189][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:44:18,689][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:44:19,194][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:44:19,694][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:44:20,195][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:44:20,698][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:44:21,198][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:44:21,702][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:44:22,207][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:44:22,711][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:44:23,217][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:44:23,720][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:44:24,228][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:44:24,730][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:44:25,233][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:44:25,740][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:44:26,241][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:44:26,754][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:44:27,256][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:44:27,757][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:44:28,259][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:44:28,764][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:44:29,266][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:44:29,770][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:44:30,275][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:44:30,781][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:44:31,285][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:44:31,786][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:44:32,285][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:44:32,791][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:44:33,293][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:44:33,795][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:44:34,296][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:44:34,796][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:44:35,296][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:44:35,795][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:44:36,293][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:44:36,792][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:44:37,292][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:44:37,791][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:44:38,288][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:44:38,791][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:44:39,291][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:44:39,788][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:44:40,287][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:44:40,786][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:44:41,287][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:44:41,787][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:44:42,290][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:44:42,789][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:44:43,291][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:44:43,793][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:44:44,293][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:44:44,795][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:44:45,297][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:44:45,800][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10122 tokens.
[2025-11-13 09:44:46,724][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.29%, Current % of VRAM taken: 60.55%, Block Peak % of device VRAM: 62.32%, ΔTime: 00:00:33
[2025-11-13 09:44:47,466][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:44:47,468][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:44:47,469][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:44:48,503][__main__][INFO] - Iteration 710 took 1m 7s (46.36% Gen, 52.12% Train). Generation: 31s, Training: 35s. Estimated remaining time: 45h 3m 9s. Estimated total time: 56h 38m 16s. Time estimates for 10 more iterations: 11m 19s, 100 more iterations: 1h 53m 16s, 500 more iterations: 9h 26m 22s.
[2025-11-13 09:44:48,505][__main__][INFO] - Starting iteration 710.
[2025-11-13 09:44:48,993][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 70 and human policies 1.
[2025-11-13 09:44:48,994][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:45:12,144][__main__][INFO] - Number of regex retries in iteration 710: 0
[2025-11-13 09:45:12,145][__main__][INFO] - agents played in iteration 710 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:45:12,978][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:13,004][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:13,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:13,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.41%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:13,055][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:45:13,055][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:45:13,945][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:45:14,404][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:45:14,914][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:45:15,414][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:45:15,914][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:45:16,413][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:45:16,913][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:45:17,413][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:45:17,916][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:45:18,416][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:45:18,916][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:45:19,425][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:45:19,926][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:45:20,428][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:45:20,933][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:45:21,436][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:45:21,954][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:45:22,454][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:45:22,954][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:45:23,459][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:45:23,959][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:45:24,461][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:45:25,754][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:45:26,527][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:45:27,032][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:45:27,561][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:45:28,060][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:45:28,572][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:45:29,074][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:45:29,577][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:45:30,076][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:45:30,579][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:45:31,085][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:45:31,585][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:45:32,087][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:45:32,588][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:45:33,090][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:45:33,597][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:45:34,095][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:45:34,599][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:45:35,101][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:45:35,603][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:45:36,108][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:45:36,610][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:45:37,113][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:45:37,616][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:45:38,119][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:45:38,627][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:45:39,146][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:45:39,647][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:45:40,164][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:45:40,664][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:45:41,168][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:45:41,675][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:45:42,179][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:45:42,679][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:45:43,181][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:45:43,680][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:45:44,179][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:45:44,680][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:45:45,180][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:45:45,689][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:45:46,192][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:45:46,693][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:45:47,191][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10077 tokens.
[2025-11-13 09:45:48,052][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.05%, Current % of VRAM taken: 60.31%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:34
[2025-11-13 09:45:48,774][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:45:48,776][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:45:48,778][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:45:50,913][__main__][INFO] - Iteration 711 took 1m 1s (37.39% Gen, 59.16% Train). Generation: 23s, Training: 36s. Estimated remaining time: 39h 59m 51s. Estimated total time: 51h 36m 0s. Time estimates for 10 more iterations: 10m 19s, 100 more iterations: 1h 43m 12s, 500 more iterations: 8h 36m 0s.
[2025-11-13 09:45:50,915][__main__][INFO] - Starting iteration 711.
[2025-11-13 09:45:51,398][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:45:51,398][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:46:22,005][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:46:29,684][__main__][INFO] - Number of regex retries in iteration 711: 1
[2025-11-13 09:46:29,684][__main__][INFO] - agents played in iteration 711 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:46:30,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:30,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:30,597][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:30,620][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.31%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:30,620][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:46:30,621][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:46:31,527][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:46:31,988][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:46:32,499][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:46:33,008][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:46:33,518][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:46:34,018][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:46:34,525][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:46:35,031][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:46:35,532][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:46:36,047][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:46:36,548][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:46:37,062][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:46:37,565][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:46:38,066][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:46:38,573][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:46:39,074][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:46:39,576][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:46:40,076][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:46:40,576][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:46:41,088][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:46:41,592][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:46:42,092][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:46:42,593][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:46:43,094][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:46:43,597][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:46:44,104][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:46:44,607][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:46:45,112][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:46:45,618][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:46:46,119][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:46:46,622][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:46:47,125][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:46:47,627][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:46:48,131][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:46:48,632][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:46:49,132][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:46:49,636][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:46:50,138][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:46:50,638][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:46:51,141][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:46:51,641][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:46:52,156][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:46:52,654][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:46:53,152][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:46:53,657][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:46:54,156][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:46:54,659][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:46:55,162][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:46:55,659][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:46:56,172][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:46:56,669][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:46:57,170][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:46:57,670][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:46:58,171][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:46:58,675][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:46:59,174][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:46:59,675][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:47:00,185][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:47:00,683][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:47:01,187][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:47:01,688][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:47:02,188][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:47:02,697][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:47:03,195][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:47:03,692][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10198 tokens.
[2025-11-13 09:47:04,531][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.21%, ΔTime: 00:00:33
[2025-11-13 09:47:05,294][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:47:05,295][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:47:05,297][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:47:06,342][__main__][INFO] - Iteration 712 took 1m 14s (51.09% Gen, 47.52% Train). Generation: 38s, Training: 35s. Estimated remaining time: 50h 49m 49s. Estimated total time: 62h 27m 13s. Time estimates for 10 more iterations: 12m 29s, 100 more iterations: 2h 4m 54s, 500 more iterations: 10h 24m 32s.
[2025-11-13 09:47:06,344][__main__][INFO] - Starting iteration 712.
[2025-11-13 09:47:06,832][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:47:06,833][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:47:21,580][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:47:31,916][__main__][INFO] - Number of regex retries in iteration 712: 1
[2025-11-13 09:47:31,916][__main__][INFO] - agents played in iteration 712 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:47:32,701][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:32,728][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:32,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:32,777][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:32,777][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:47:32,778][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:47:33,604][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:47:34,084][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:47:34,588][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:47:35,089][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:47:35,591][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:47:36,091][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:47:36,596][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:47:37,095][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:47:37,595][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:47:38,100][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:47:38,599][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:47:39,099][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:47:39,601][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:47:40,109][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:47:40,621][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:47:41,123][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:47:41,626][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:47:42,129][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:47:42,631][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:47:43,135][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:47:43,639][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:47:44,141][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:47:44,649][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:47:45,153][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:47:45,657][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:47:46,166][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:47:48,034][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:47:48,722][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:47:49,225][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:47:49,728][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:47:50,234][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:47:50,733][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:47:51,239][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:47:51,739][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:47:52,239][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:47:52,751][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:47:53,254][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:47:53,771][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:47:54,278][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:47:54,777][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:47:55,283][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:47:55,785][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:47:56,292][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:47:56,793][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:47:57,294][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:47:57,796][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:47:58,297][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:47:58,801][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:47:59,305][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:47:59,806][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:48:00,308][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:48:00,811][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:48:01,312][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:48:01,814][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:48:02,314][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:48:02,816][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:48:03,312][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:48:03,809][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:48:04,311][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:48:04,810][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:48:05,307][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:48:05,805][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:48:06,302][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:48:06,804][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:48:07,301][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10110 tokens.
[2025-11-13 09:48:08,214][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:34
[2025-11-13 09:48:08,860][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:48:08,862][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:48:08,864][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:48:09,924][__main__][INFO] - Iteration 713 took 1m 3s (39.76% Gen, 58.56% Train). Generation: 25s, Training: 36s. Estimated remaining time: 40h 56m 9s. Estimated total time: 52h 34m 37s. Time estimates for 10 more iterations: 10m 30s, 100 more iterations: 1h 45m 9s, 500 more iterations: 8h 45m 46s.
[2025-11-13 09:48:09,927][__main__][INFO] - Starting iteration 713.
[2025-11-13 09:48:10,417][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:48:10,417][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:48:38,307][__main__][INFO] - Number of regex retries in iteration 713: 0
[2025-11-13 09:48:38,307][__main__][INFO] - agents played in iteration 713 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:48:39,275][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:39,298][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:39,323][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:39,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:39,347][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:48:39,348][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:48:40,246][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:48:40,703][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:48:41,205][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:48:41,715][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:48:42,216][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:48:42,716][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:48:43,216][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:48:43,719][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:48:44,220][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:48:44,718][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:48:45,217][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:48:45,720][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:48:46,232][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:48:46,733][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:48:47,235][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:48:47,733][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:48:48,233][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:48:48,733][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:48:49,233][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:48:49,736][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:48:50,238][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:48:50,740][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:48:51,240][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:48:51,740][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:48:52,239][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:48:52,741][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:48:53,241][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:48:53,742][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:48:54,247][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:48:54,747][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:48:55,249][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:48:55,751][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:48:56,254][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:48:56,767][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:48:57,269][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:48:57,778][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:48:58,281][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:48:58,780][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:48:59,282][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:48:59,782][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:49:00,282][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:49:00,784][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:49:01,288][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:49:01,802][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:49:02,306][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:49:02,809][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:49:03,312][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:49:03,815][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:49:04,319][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:49:04,822][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:49:05,320][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:49:05,820][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:49:06,323][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:49:06,823][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:49:07,328][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:49:07,828][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:49:08,331][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:49:08,833][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:49:09,337][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:49:09,841][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:49:10,344][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:49:10,847][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:49:11,351][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:49:11,854][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:49:12,357][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10019 tokens.
[2025-11-13 09:49:13,245][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.11%, ΔTime: 00:00:33
[2025-11-13 09:49:14,035][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:49:14,037][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:49:14,040][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:49:15,098][__main__][INFO] - Iteration 714 took 1m 4s (43.12% Gen, 55.24% Train). Generation: 27s, Training: 35s. Estimated remaining time: 42h 14m 31s. Estimated total time: 53h 54m 4s. Time estimates for 10 more iterations: 10m 46s, 100 more iterations: 1h 47m 48s, 500 more iterations: 8h 59m 0s.
[2025-11-13 09:49:15,100][__main__][INFO] - Starting iteration 714.
[2025-11-13 09:49:15,581][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:49:15,582][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:49:52,681][__main__][INFO] - Number of regex retries in iteration 714: 0
[2025-11-13 09:49:52,682][__main__][INFO] - agents played in iteration 714 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:49:53,457][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:53,484][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:53,510][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:53,533][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:53,534][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:49:53,535][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:49:54,362][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:49:54,818][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:49:55,327][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:49:55,828][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:49:56,332][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:49:56,838][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:49:57,340][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:49:57,838][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:49:58,337][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:49:58,835][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:49:59,334][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:49:59,842][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:50:00,347][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:50:00,853][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:50:01,355][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:50:01,862][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:50:02,369][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:50:02,873][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:50:03,950][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:50:05,119][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:50:05,620][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:50:06,134][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:50:06,634][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:50:07,135][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:50:07,634][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:50:08,139][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:50:08,643][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:50:09,143][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:50:09,647][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:50:10,149][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:50:10,651][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:50:11,153][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:50:11,655][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:50:12,160][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:50:12,667][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:50:13,171][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:50:13,676][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:50:14,180][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:50:14,704][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:50:15,213][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:50:15,720][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:50:16,223][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:50:16,731][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:50:17,235][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:50:17,735][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:50:18,242][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:50:18,741][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:50:19,245][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:50:19,744][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:50:20,245][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:50:20,752][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:50:21,256][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:50:21,762][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:50:22,264][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:50:22,768][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:50:23,272][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:50:23,775][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:50:24,273][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:50:24,770][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:50:25,268][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:50:25,766][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:50:26,264][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:50:26,761][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:50:27,260][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:50:27,760][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10202 tokens.
[2025-11-13 09:50:28,674][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.25%, Current % of VRAM taken: 60.51%, Block Peak % of device VRAM: 62.51%, ΔTime: 00:00:34
[2025-11-13 09:50:29,324][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:50:29,326][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:50:29,328][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:50:30,250][__main__][INFO] - Iteration 715 took 1m 14s (49.68% Gen, 49.08% Train). Generation: 37s, Training: 36s. Estimated remaining time: 50h 32m 39s. Estimated total time: 62h 13m 28s. Time estimates for 10 more iterations: 12m 26s, 100 more iterations: 2h 4m 26s, 500 more iterations: 10h 22m 14s.
[2025-11-13 09:50:30,253][__main__][INFO] - Starting iteration 715.
[2025-11-13 09:50:30,732][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:50:30,733][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:51:04,335][__main__][INFO] - Number of regex retries in iteration 715: 0
[2025-11-13 09:51:04,335][__main__][INFO] - agents played in iteration 715 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:51:05,169][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.44%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:05,196][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.44%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:05,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.44%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:05,245][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.44%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:05,246][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:51:05,247][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:51:06,089][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:51:06,547][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:51:07,051][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:51:07,552][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:51:08,053][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:51:08,554][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:51:09,063][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:51:09,570][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:51:10,073][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:51:10,571][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:51:11,076][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:51:11,580][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:51:12,083][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:51:12,589][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:51:13,099][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:51:13,602][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:51:14,126][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:51:14,632][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:51:15,136][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:51:15,641][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:51:16,145][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:51:16,651][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:51:17,154][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:51:17,659][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:51:18,166][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:51:18,666][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:51:19,168][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:51:19,668][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:51:20,174][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:51:20,679][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:51:21,182][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:51:21,685][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:51:22,199][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:51:22,698][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:51:23,198][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:51:23,697][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:51:24,196][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:51:24,698][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:51:25,201][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:51:25,703][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:51:26,211][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:51:26,710][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:51:27,211][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:51:27,717][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:51:28,215][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:51:28,718][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:51:29,219][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:51:29,723][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:51:30,224][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:51:30,724][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:51:31,230][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:51:31,732][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:51:32,236][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:51:32,738][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:51:33,241][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:51:33,742][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:51:34,240][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:51:34,743][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:51:35,246][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:51:35,742][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:51:36,245][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:51:36,747][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:51:37,245][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:51:37,743][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:51:38,242][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10153 tokens.
[2025-11-13 09:51:39,104][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.12%, Current % of VRAM taken: 60.38%, Block Peak % of device VRAM: 62.39%, ΔTime: 00:00:33
[2025-11-13 09:51:39,852][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:51:39,854][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:51:39,856][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:51:40,856][__main__][INFO] - Iteration 716 took 1m 10s (47.92% Gen, 50.65% Train). Generation: 33s, Training: 35s. Estimated remaining time: 46h 44m 12s. Estimated total time: 58h 26m 12s. Time estimates for 10 more iterations: 11m 41s, 100 more iterations: 1h 56m 52s, 500 more iterations: 9h 44m 22s.
[2025-11-13 09:51:40,858][__main__][INFO] - Starting iteration 716.
[2025-11-13 09:51:41,362][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:51:41,363][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:52:08,020][__main__][INFO] - Number of regex retries in iteration 716: 0
[2025-11-13 09:52:08,020][__main__][INFO] - agents played in iteration 716 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:52:08,797][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:08,824][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:08,851][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:08,874][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:08,875][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:52:08,875][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:52:09,730][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:52:10,190][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:52:10,695][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:52:11,197][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:52:11,701][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:52:12,197][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:52:12,693][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:52:13,192][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:52:13,690][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:52:14,189][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:52:14,690][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:52:15,187][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:52:15,686][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:52:16,183][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:52:16,681][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:52:17,180][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:52:17,679][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:52:18,177][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:52:18,690][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:52:19,189][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:52:19,687][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:52:20,184][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:52:20,681][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:52:21,180][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:52:21,679][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:52:22,180][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:52:22,691][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:52:23,191][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:52:23,694][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:52:24,195][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:52:24,693][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:52:25,201][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:52:25,707][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:52:26,209][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:52:26,714][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:52:27,217][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:52:27,727][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:52:28,226][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:52:28,725][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:52:29,224][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:52:29,723][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:52:30,224][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:52:30,726][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:52:31,226][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:52:31,733][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:52:32,234][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:52:32,733][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:52:33,234][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:52:33,736][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:52:34,234][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:52:34,731][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:52:35,230][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:52:35,728][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:52:36,224][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:52:36,722][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:52:37,217][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:52:37,714][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:52:38,221][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:52:38,720][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:52:39,220][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:52:39,718][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:52:40,217][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:52:40,718][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:52:41,217][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:52:41,715][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10055 tokens.
[2025-11-13 09:52:42,577][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.07%, ΔTime: 00:00:32
[2025-11-13 09:52:43,326][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:52:43,328][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:52:43,329][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:52:44,617][__main__][INFO] - Iteration 717 took 1m 3s (42.14% Gen, 55.82% Train). Generation: 26s, Training: 35s. Estimated remaining time: 40h 59m 44s. Estimated total time: 52h 42m 47s. Time estimates for 10 more iterations: 10m 32s, 100 more iterations: 1h 45m 25s, 500 more iterations: 8h 47m 7s.
[2025-11-13 09:52:44,619][__main__][INFO] - Starting iteration 717.
[2025-11-13 09:52:45,115][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:52:45,116][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:53:10,793][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:53:11,725][__main__][INFO] - Number of regex retries in iteration 717: 1
[2025-11-13 09:53:11,726][__main__][INFO] - agents played in iteration 717 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:53:12,714][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:12,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:12,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:12,811][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:12,812][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:53:12,812][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:53:13,665][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:53:14,118][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:53:14,621][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:53:15,137][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:53:15,635][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:53:16,143][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:53:16,641][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:53:17,139][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:53:17,643][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:53:18,146][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:53:18,645][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:53:19,147][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:53:19,648][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:53:20,156][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:53:20,652][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:53:21,154][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:53:21,660][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:53:22,160][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:53:22,662][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:53:23,165][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:53:23,662][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:53:24,168][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:53:24,668][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:53:25,173][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:53:25,676][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:53:26,178][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:53:26,685][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:53:27,186][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:53:27,690][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:53:28,208][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:53:28,709][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:53:29,211][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:53:29,713][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:53:30,219][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:53:30,738][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:53:31,260][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:53:31,772][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:53:32,275][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:53:32,780][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:53:33,284][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:53:33,786][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:53:34,285][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:53:34,786][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:53:35,288][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:53:35,789][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:53:36,300][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:53:36,804][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:53:37,309][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:53:37,812][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:53:38,313][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:53:38,830][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:53:39,330][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:53:39,834][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:53:40,336][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:53:40,839][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:53:41,340][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:53:41,844][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:53:42,344][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:53:42,846][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:53:43,345][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:53:43,845][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:53:44,348][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:53:44,848][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:53:45,348][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:53:45,843][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10070 tokens.
[2025-11-13 09:53:46,690][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.08%, Current % of VRAM taken: 60.34%, Block Peak % of device VRAM: 62.43%, ΔTime: 00:00:33
[2025-11-13 09:53:47,356][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:53:47,358][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:53:47,359][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:53:48,372][__main__][INFO] - Iteration 718 took 1m 3s (42.07% Gen, 56.33% Train). Generation: 26s, Training: 35s. Estimated remaining time: 40h 58m 45s. Estimated total time: 52h 42m 52s. Time estimates for 10 more iterations: 10m 32s, 100 more iterations: 1h 45m 25s, 500 more iterations: 8h 47m 8s.
[2025-11-13 09:53:48,374][__main__][INFO] - Starting iteration 718.
[2025-11-13 09:53:48,844][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:53:48,844][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:54:17,428][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:54:20,797][__main__][INFO] - Number of regex retries in iteration 718: 1
[2025-11-13 09:54:20,798][__main__][INFO] - agents played in iteration 718 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:54:21,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:21,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:21,698][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:21,721][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.38%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:21,721][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:54:21,722][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:54:22,553][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:54:23,005][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:54:23,510][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:54:24,006][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:54:24,502][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:54:25,015][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:54:25,516][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:54:26,011][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:54:26,507][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:54:27,004][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:54:27,509][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:54:28,009][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:54:28,505][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:54:29,003][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:54:29,503][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:54:29,998][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:54:30,498][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:54:30,994][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:54:31,492][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:54:31,996][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:54:32,497][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:54:33,009][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:54:33,509][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:54:34,027][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:54:34,526][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:54:35,035][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:54:35,552][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:54:36,057][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:54:36,561][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:54:37,065][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:54:37,569][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:54:38,073][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:54:38,574][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:54:39,077][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:54:39,582][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:54:40,088][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:54:40,593][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:54:41,103][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:54:41,606][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:54:42,115][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:54:42,620][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:54:43,123][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:54:43,630][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:54:44,132][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:54:44,637][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:54:45,143][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:54:45,647][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:54:46,155][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:54:46,658][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:54:47,162][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:54:47,665][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:54:48,170][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:54:48,671][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:54:49,174][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:54:49,677][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:54:50,182][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:54:50,684][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:54:51,185][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:54:51,687][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:54:52,191][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:54:52,695][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:54:53,193][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:54:53,695][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:54:54,197][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:54:54,699][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10169 tokens.
[2025-11-13 09:54:55,680][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.25%, ΔTime: 00:00:33
[2025-11-13 09:54:56,450][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:54:56,453][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:54:56,454][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:54:57,445][__main__][INFO] - Iteration 719 took 1m 8s (46.58% Gen, 51.98% Train). Generation: 31s, Training: 35s. Estimated remaining time: 45h 24m 49s. Estimated total time: 57h 10m 5s. Time estimates for 10 more iterations: 11m 26s, 100 more iterations: 1h 54m 20s, 500 more iterations: 9h 31m 40s.
[2025-11-13 09:54:57,447][__main__][INFO] - Starting iteration 719.
[2025-11-13 09:54:57,921][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:54:57,922][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:55:25,924][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:55:29,701][__main__][INFO] - Number of regex retries in iteration 719: 1
[2025-11-13 09:55:29,701][__main__][INFO] - agents played in iteration 719 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:55:30,546][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:30,569][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:30,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:30,613][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:30,613][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:55:30,614][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:55:31,440][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:55:31,901][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:55:32,408][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:55:32,910][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:55:33,409][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:55:33,904][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:55:34,412][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:55:34,910][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:55:35,414][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:55:35,915][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:55:36,412][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:55:36,913][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:55:37,410][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:55:37,910][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:55:38,405][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:55:38,909][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:55:39,411][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:55:39,910][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:55:40,419][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:55:40,927][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:55:41,429][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:55:41,931][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:55:42,434][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:55:42,929][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:55:43,453][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:55:43,954][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:55:44,456][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:55:44,957][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:55:45,457][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:55:45,968][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:55:46,473][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:55:46,978][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:55:47,480][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:55:47,981][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:55:48,479][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:55:48,982][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:55:49,481][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:55:49,980][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:55:50,479][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:55:50,978][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:55:51,477][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:55:51,982][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:55:52,496][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:55:53,003][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:55:53,505][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:55:54,012][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:55:54,516][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:55:55,019][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:55:55,522][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:55:56,023][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:55:56,531][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:55:57,033][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:55:57,535][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:55:58,042][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:55:58,544][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:55:59,058][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:55:59,559][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:56:00,061][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:56:00,566][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:56:01,066][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:56:01,573][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:56:02,077][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:56:02,581][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:56:03,084][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:56:03,585][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10071 tokens.
[2025-11-13 09:56:04,559][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.10%, Current % of VRAM taken: 60.36%, Block Peak % of device VRAM: 62.16%, ΔTime: 00:00:33
[2025-11-13 09:56:05,203][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:56:05,205][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:56:05,207][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:56:06,119][__main__][INFO] - Iteration 720 took 1m 8s (46.60% Gen, 52.06% Train). Generation: 31s, Training: 35s. Estimated remaining time: 45h 3m 30s. Estimated total time: 56h 49m 55s. Time estimates for 10 more iterations: 11m 21s, 100 more iterations: 1h 53m 39s, 500 more iterations: 9h 28m 19s.
[2025-11-13 09:56:06,121][__main__][INFO] - Starting iteration 720.
[2025-11-13 09:56:06,600][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 71 and human policies 1.
[2025-11-13 09:56:06,601][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:56:26,871][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:56:32,906][__main__][INFO] - Number of regex retries in iteration 720: 1
[2025-11-13 09:56:32,907][__main__][INFO] - agents played in iteration 720 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:56:33,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:33,760][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:33,787][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:33,811][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.33%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:33,812][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:56:33,813][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:56:34,655][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:56:35,114][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:56:35,617][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:56:36,115][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:56:36,613][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:56:37,111][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:56:37,612][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:56:38,115][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:56:38,618][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:56:39,114][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:56:39,613][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:56:40,112][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:56:40,611][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:56:41,108][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:56:41,605][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:56:42,104][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:56:42,603][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:56:43,100][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:56:43,598][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:56:44,096][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:56:44,600][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:56:45,108][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:56:45,609][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:56:46,117][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:56:46,620][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:56:47,120][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:56:47,622][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:56:48,121][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:56:48,622][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:56:49,128][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:56:49,632][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:56:50,148][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:56:50,652][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:56:51,153][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:56:51,654][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:56:52,158][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:56:52,679][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:56:53,181][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:56:53,682][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:56:54,183][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:56:54,684][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:56:55,188][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:56:55,693][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:56:56,196][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:56:56,698][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:56:57,199][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:56:57,706][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:56:58,210][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:56:58,714][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:56:59,222][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:56:59,721][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:57:00,225][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:57:00,725][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:57:01,226][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:57:01,727][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:57:02,229][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:57:02,731][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:57:03,234][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:57:03,737][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:57:04,237][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:57:04,753][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:57:05,254][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:57:05,771][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:57:06,274][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:57:06,777][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10156 tokens.
[2025-11-13 09:57:07,760][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.22%, ΔTime: 00:00:33
[2025-11-13 09:57:08,514][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:57:08,516][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:57:08,518][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:57:10,451][__main__][INFO] - Iteration 721 took 1m 3s (41.20% Gen, 55.77% Train). Generation: 26s, Training: 35s. Estimated remaining time: 41h 25m 5s. Estimated total time: 53h 12m 34s. Time estimates for 10 more iterations: 10m 38s, 100 more iterations: 1h 46m 25s, 500 more iterations: 8h 52m 5s.
[2025-11-13 09:57:10,453][__main__][INFO] - Starting iteration 721.
[2025-11-13 09:57:10,941][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 09:57:10,942][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:57:35,431][__main__][INFO] - Number of regex retries in iteration 721: 0
[2025-11-13 09:57:35,433][__main__][INFO] - agents played in iteration 721 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:57:36,499][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:36,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:36,549][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:36,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:36,572][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:57:36,573][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:57:37,404][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:57:37,863][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:57:38,369][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:57:38,871][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:57:39,371][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:57:39,871][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:57:40,373][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:57:40,874][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:57:41,381][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:57:41,876][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:57:42,375][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:57:42,889][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:57:43,385][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:57:43,883][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:57:44,379][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:57:44,875][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:57:45,390][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:57:45,889][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:57:46,389][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:57:46,891][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:57:47,388][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:57:47,885][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:57:48,382][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:57:48,885][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:57:49,394][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:57:49,892][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:57:50,394][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:57:50,888][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:57:51,390][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:57:51,890][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:57:52,394][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:57:52,893][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:57:53,396][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:57:53,894][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:57:54,393][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:57:54,892][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:57:55,395][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:57:55,903][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:57:56,408][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:57:56,912][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:57:57,415][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:57:57,911][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:57:58,415][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:57:58,915][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:57:59,414][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:57:59,915][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:58:00,416][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:58:00,913][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:58:01,414][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:58:01,914][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:58:02,417][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:58:02,916][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:58:03,417][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:58:03,917][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:58:04,417][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:58:04,918][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:58:05,418][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:58:05,926][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:58:06,431][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:58:06,935][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:58:07,438][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:58:07,940][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:58:08,446][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:58:08,958][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:58:09,461][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10055 tokens.
[2025-11-13 09:58:10,433][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.19%, ΔTime: 00:00:33
[2025-11-13 09:58:11,077][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:58:11,078][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:58:11,080][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:58:12,062][__main__][INFO] - Iteration 722 took 1m 1s (40.07% Gen, 58.32% Train). Generation: 24s, Training: 35s. Estimated remaining time: 39h 7m 33s. Estimated total time: 50h 56m 3s. Time estimates for 10 more iterations: 10m 11s, 100 more iterations: 1h 41m 52s, 500 more iterations: 8h 29m 20s.
[2025-11-13 09:58:12,064][__main__][INFO] - Starting iteration 722.
[2025-11-13 09:58:12,539][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 09:58:12,540][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:58:30,007][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 0 hats, 0 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 09:58:39,995][__main__][INFO] - Number of regex retries in iteration 722: 1
[2025-11-13 09:58:39,996][__main__][INFO] - agents played in iteration 722 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:58:40,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:40,860][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:40,886][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:40,910][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.36%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:40,910][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:58:40,911][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:58:41,722][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:58:42,193][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:58:42,694][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:58:43,190][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:58:43,685][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:58:44,182][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:58:44,690][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:58:45,190][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:58:45,685][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:58:46,193][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:58:46,692][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:58:47,188][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:58:47,689][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:58:48,187][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:58:48,685][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:58:49,183][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:58:49,679][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:58:50,197][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:58:50,695][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:58:51,197][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:58:51,697][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:58:52,194][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:58:52,700][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:58:53,198][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:58:53,694][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:58:54,200][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:58:54,696][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:58:55,194][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:58:55,692][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:58:56,191][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:58:56,694][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:58:57,193][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:58:57,693][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:58:58,198][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:58:58,695][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 09:58:59,191][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 09:58:59,688][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 09:59:00,185][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 09:59:00,690][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 09:59:01,185][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 09:59:01,686][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 09:59:02,198][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 09:59:02,694][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 09:59:03,196][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 09:59:03,695][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 09:59:04,191][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 09:59:04,701][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 09:59:05,200][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 09:59:05,696][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 09:59:06,193][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 09:59:06,697][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 09:59:07,199][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 09:59:07,699][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 09:59:08,199][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 09:59:08,709][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 09:59:09,210][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 09:59:09,710][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 09:59:10,212][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 09:59:10,715][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 09:59:11,220][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 09:59:11,722][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 09:59:12,226][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 09:59:12,727][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 09:59:13,228][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 09:59:13,731][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10021 tokens.
[2025-11-13 09:59:14,693][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.11%, Current % of VRAM taken: 60.37%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:32
[2025-11-13 09:59:15,463][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:59:15,465][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:59:15,467][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:59:16,479][__main__][INFO] - Iteration 723 took 1m 3s (42.94% Gen, 55.47% Train). Generation: 27s, Training: 35s. Estimated remaining time: 41h 27m 28s. Estimated total time: 53h 17m 3s. Time estimates for 10 more iterations: 10m 39s, 100 more iterations: 1h 46m 34s, 500 more iterations: 8h 52m 50s.
[2025-11-13 09:59:16,482][__main__][INFO] - Starting iteration 723.
[2025-11-13 09:59:16,956][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 09:59:16,957][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:59:41,025][__main__][INFO] - Number of regex retries in iteration 723: 0
[2025-11-13 09:59:41,026][__main__][INFO] - agents played in iteration 723 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 09:59:41,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:41,831][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:41,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:41,880][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:41,880][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:59:41,881][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:59:42,689][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 09:59:43,146][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 09:59:43,647][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 09:59:44,146][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 09:59:44,645][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 09:59:45,141][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 09:59:45,637][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 09:59:46,132][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 09:59:46,629][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 09:59:47,126][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 09:59:47,624][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 09:59:48,123][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 09:59:48,624][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 09:59:49,119][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 09:59:49,616][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 09:59:50,112][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 09:59:50,612][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 09:59:51,111][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 09:59:51,609][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 09:59:52,108][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 09:59:52,605][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 09:59:53,126][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 09:59:53,627][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 09:59:54,129][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 09:59:54,628][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 09:59:55,123][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 09:59:55,623][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 09:59:56,124][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 09:59:56,620][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 09:59:57,116][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 09:59:57,614][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 09:59:58,110][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 09:59:58,616][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 09:59:59,111][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 09:59:59,608][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 10:00:00,103][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 10:00:00,599][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 10:00:01,101][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 10:00:01,596][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 10:00:02,091][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 10:00:02,599][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 10:00:03,094][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 10:00:03,590][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 10:00:04,089][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 10:00:04,585][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 10:00:05,080][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 10:00:05,577][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 10:00:06,073][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 10:00:06,585][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 10:00:07,081][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 10:00:07,579][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 10:00:08,076][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 10:00:08,573][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 10:00:09,074][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 10:00:09,575][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 10:00:10,078][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 10:00:10,590][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 10:00:11,364][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 10:00:11,923][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 10:00:12,425][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 10:00:12,930][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 10:00:13,434][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 10:00:13,935][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 10:00:14,443][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 10:00:14,944][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10049 tokens.
[2025-11-13 10:00:15,931][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.17%, Current % of VRAM taken: 60.43%, Block Peak % of device VRAM: 62.30%, ΔTime: 00:00:33
[2025-11-13 10:00:16,613][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:00:16,617][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:00:16,619][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:00:17,813][__main__][INFO] - Iteration 724 took 1m 0s (39.55% Gen, 58.49% Train). Generation: 24s, Training: 35s. Estimated remaining time: 38h 52m 16s. Estimated total time: 50h 42m 53s. Time estimates for 10 more iterations: 10m 8s, 100 more iterations: 1h 41m 25s, 500 more iterations: 8h 27m 8s.
[2025-11-13 10:00:17,815][__main__][INFO] - Starting iteration 724.
[2025-11-13 10:00:18,301][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 10:00:18,302][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:00:51,650][__main__][INFO] - Number of regex retries in iteration 724: 0
[2025-11-13 10:00:51,650][__main__][INFO] - agents played in iteration 724 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 10:00:52,498][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:52,525][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:52,552][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:52,575][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.40%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:52,575][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:00:52,577][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:00:53,385][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 10:00:53,843][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 10:00:54,343][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 10:00:54,840][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 10:00:55,335][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 10:00:55,832][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 10:00:56,330][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 10:00:56,825][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 10:00:57,321][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 10:00:57,832][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 10:00:58,331][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 10:00:58,833][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 10:00:59,332][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 10:00:59,831][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 10:01:00,335][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 10:01:00,833][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 10:01:01,332][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 10:01:01,835][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 10:01:02,334][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 10:01:02,833][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 10:01:03,327][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 10:01:03,822][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 10:01:04,326][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 10:01:04,823][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 10:01:05,322][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 10:01:05,817][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 10:01:06,314][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 10:01:06,818][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 10:01:07,314][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 10:01:07,813][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 10:01:08,321][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 10:01:08,817][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 10:01:09,313][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 10:01:09,813][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 10:01:10,309][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 10:01:10,817][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 10:01:11,314][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 10:01:11,811][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 10:01:12,309][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 10:01:12,810][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 10:01:13,306][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 10:01:13,807][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 10:01:14,306][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 10:01:14,813][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 10:01:15,309][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 10:01:15,806][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 10:01:16,308][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 10:01:16,807][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 10:01:17,303][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 10:01:17,801][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 10:01:18,303][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 10:01:18,824][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 10:01:19,324][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 10:01:19,824][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 10:01:20,325][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 10:01:20,827][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 10:01:21,335][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 10:01:21,839][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 10:01:22,340][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 10:01:22,846][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 10:01:23,347][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 10:01:23,851][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 10:01:24,353][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 10:01:24,857][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 10:01:25,363][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10097 tokens.
[2025-11-13 10:01:26,332][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.00%, Current % of VRAM taken: 60.25%, Block Peak % of device VRAM: 62.24%, ΔTime: 00:00:32
[2025-11-13 10:01:27,097][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:01:27,098][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:01:27,100][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:01:28,114][__main__][INFO] - Iteration 725 took 1m 9s (47.77% Gen, 50.78% Train). Generation: 33s, Training: 35s. Estimated remaining time: 46h 18m 54s. Estimated total time: 58h 10m 41s. Time estimates for 10 more iterations: 11m 38s, 100 more iterations: 1h 56m 21s, 500 more iterations: 9h 41m 46s.
[2025-11-13 10:01:28,116][__main__][INFO] - Starting iteration 725.
[2025-11-13 10:01:28,596][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 10:01:28,596][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:01:56,090][__main__][INFO] - Number of regex retries in iteration 725: 0
[2025-11-13 10:01:56,091][__main__][INFO] - agents played in iteration 725 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 10:01:56,901][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:56,926][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:56,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:56,974][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.27%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:56,974][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:01:56,975][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:01:57,783][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 10:01:58,240][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 10:01:58,743][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 10:01:59,242][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 10:01:59,742][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 10:02:00,241][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 10:02:00,742][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 10:02:01,238][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 10:02:01,735][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 10:02:02,240][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 10:02:02,738][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 10:02:03,237][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 10:02:03,733][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 10:02:04,232][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 10:02:04,735][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 10:02:05,232][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 10:02:05,732][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 10:02:06,230][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 10:02:06,727][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 10:02:07,225][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 10:02:07,721][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 10:02:08,222][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 10:02:08,732][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 10:02:09,230][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 10:02:09,726][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 10:02:10,225][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 10:02:10,726][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 10:02:11,223][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 10:02:11,721][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 10:02:12,225][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 10:02:12,734][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 10:02:13,230][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 10:02:13,728][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 10:02:14,228][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 10:02:14,727][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 10:02:15,238][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 10:02:15,737][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 10:02:16,261][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 10:02:16,760][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 10:02:17,259][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 10:02:17,762][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 10:02:18,262][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 10:02:18,764][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 10:02:19,270][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 10:02:19,775][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 10:02:20,278][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 10:02:20,780][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 10:02:21,281][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 10:02:21,783][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 10:02:22,284][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 10:02:22,785][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 10:02:23,289][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 10:02:23,793][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 10:02:24,294][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 10:02:24,795][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 10:02:25,298][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 10:02:25,807][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 10:02:26,313][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 10:02:26,815][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 10:02:27,443][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 10:02:28,079][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 10:02:28,583][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 10:02:29,085][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 10:02:29,588][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 10:02:30,091][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10100 tokens.
[2025-11-13 10:02:31,117][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.07%, Current % of VRAM taken: 60.33%, Block Peak % of device VRAM: 62.41%, ΔTime: 00:00:33
[2025-11-13 10:02:31,794][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:02:31,796][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:02:31,798][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:02:32,709][__main__][INFO] - Iteration 726 took 1m 4s (42.88% Gen, 55.69% Train). Generation: 27s, Training: 35s. Estimated remaining time: 41h 32m 51s. Estimated total time: 53h 25m 42s. Time estimates for 10 more iterations: 10m 41s, 100 more iterations: 1h 46m 51s, 500 more iterations: 8h 54m 17s.
[2025-11-13 10:02:32,713][__main__][INFO] - Starting iteration 726.
[2025-11-13 10:02:33,191][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 10:02:33,191][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:03:08,212][__main__][INFO] - Number of regex retries in iteration 726: 0
[2025-11-13 10:03:08,213][__main__][INFO] - agents played in iteration 726 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 10:03:09,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:09,245][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:09,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:09,292][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.34%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:09,293][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:03:09,294][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:03:10,109][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 10:03:10,570][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 10:03:11,073][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 10:03:11,579][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 10:03:12,086][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 10:03:12,588][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 10:03:13,088][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 10:03:13,589][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 10:03:14,092][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 10:03:14,590][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 10:03:15,088][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 10:03:15,585][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 10:03:16,084][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 10:03:16,581][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 10:03:17,079][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 10:03:17,574][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 10:03:18,072][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 10:03:18,575][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 10:03:19,077][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 10:03:19,579][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 10:03:20,079][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 10:03:20,582][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 10:03:21,084][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 10:03:21,586][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 10:03:22,091][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 10:03:22,595][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 10:03:23,095][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 10:03:23,598][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 10:03:24,098][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 10:03:24,598][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 10:03:25,101][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 10:03:25,603][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 10:03:26,107][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 10:03:26,610][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 10:03:27,112][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 10:03:27,617][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 10:03:28,120][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 10:03:28,623][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 10:03:29,129][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 10:03:29,631][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 10:03:30,135][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 10:03:30,637][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 10:03:31,140][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 10:03:31,649][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 10:03:32,152][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 10:03:32,658][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 10:03:33,160][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 10:03:33,669][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 10:03:34,176][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 10:03:34,682][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 10:03:35,184][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 10:03:35,687][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 10:03:36,187][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 10:03:36,691][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 10:03:37,191][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 10:03:37,691][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 10:03:38,201][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 10:03:38,706][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 10:03:39,210][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 10:03:39,729][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 10:03:40,237][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 10:03:40,748][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 10:03:41,248][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 10:03:41,755][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 10:03:42,267][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10099 tokens.
[2025-11-13 10:03:43,208][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.23%, Current % of VRAM taken: 60.49%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:33
[2025-11-13 10:03:43,990][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:03:43,992][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:03:43,994][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:03:45,171][__main__][INFO] - Iteration 727 took 1m 11s (48.65% Gen, 49.71% Train). Generation: 35s, Training: 35s. Estimated remaining time: 48h 4m 57s. Estimated total time: 59h 59m 1s. Time estimates for 10 more iterations: 11m 59s, 100 more iterations: 1h 59m 58s, 500 more iterations: 9h 59m 50s.
[2025-11-13 10:03:45,174][__main__][INFO] - Starting iteration 727.
[2025-11-13 10:03:45,644][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 10:03:45,645][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:04:10,466][__main__][INFO] - Number of regex retries in iteration 727: 0
[2025-11-13 10:04:10,466][__main__][INFO] - agents played in iteration 727 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 10:04:11,400][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:11,427][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:11,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:11,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.39%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:11,479][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:04:11,480][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:04:12,292][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 10:04:12,746][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 10:04:13,249][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 10:04:13,746][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 10:04:14,252][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 10:04:14,768][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 10:04:15,267][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 10:04:15,762][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 10:04:16,260][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 10:04:16,754][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 10:04:17,249][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 10:04:17,745][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 10:04:18,241][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 10:04:18,758][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 10:04:19,252][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 10:04:19,747][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 10:04:20,245][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 10:04:20,743][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 10:04:21,256][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 10:04:21,760][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 10:04:22,261][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 10:04:22,763][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 10:04:23,265][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 10:04:23,773][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 10:04:24,273][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 10:04:24,774][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 10:04:25,279][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 10:04:25,779][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 10:04:26,282][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 10:04:26,783][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 10:04:27,283][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 10:04:27,784][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 10:04:28,283][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 10:04:28,784][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 10:04:29,285][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 10:04:29,787][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 10:04:30,291][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 10:04:30,793][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 10:04:31,295][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 10:04:31,799][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 10:04:32,302][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 10:04:32,809][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 10:04:33,310][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 10:04:33,817][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 10:04:34,317][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 10:04:34,818][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 10:04:35,318][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 10:04:35,827][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 10:04:36,327][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 10:04:36,830][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 10:04:37,337][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 10:04:37,838][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 10:04:39,071][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 10:04:39,608][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 10:04:40,111][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 10:04:40,657][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 10:04:41,170][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 10:04:41,674][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 10:04:42,178][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 10:04:42,679][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 10:04:43,180][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 10:04:43,685][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 10:04:44,189][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 10:04:44,694][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 10:04:45,197][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10059 tokens.
[2025-11-13 10:04:46,197][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.09%, Current % of VRAM taken: 60.35%, Block Peak % of device VRAM: 62.09%, ΔTime: 00:00:33
[2025-11-13 10:04:46,860][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:04:46,862][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:04:46,864][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:04:47,800][__main__][INFO] - Iteration 728 took 1m 2s (39.93% Gen, 58.56% Train). Generation: 24s, Training: 36s. Estimated remaining time: 39h 52m 44s. Estimated total time: 51h 47m 50s. Time estimates for 10 more iterations: 10m 21s, 100 more iterations: 1h 43m 35s, 500 more iterations: 8h 37m 58s.
[2025-11-13 10:04:47,804][__main__][INFO] - Starting iteration 728.
[2025-11-13 10:04:48,288][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 10:04:48,288][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:05:19,048][__main__][INFO] - Number of regex retries in iteration 728: 0
[2025-11-13 10:05:19,048][__main__][INFO] - agents played in iteration 728 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 10:05:20,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:20,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:20,057][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:20,079][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.32%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:20,080][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:05:20,081][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:05:20,915][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 10:05:21,375][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 10:05:21,878][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 10:05:22,381][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 10:05:22,877][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 10:05:23,374][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 10:05:23,871][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 10:05:24,389][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 10:05:24,887][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 10:05:25,385][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 10:05:25,887][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 10:05:26,384][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 10:05:26,887][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 10:05:27,389][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 10:05:27,893][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 10:05:28,395][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 10:05:28,898][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 10:05:29,401][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 10:05:29,898][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 10:05:30,397][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 10:05:30,899][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 10:05:31,401][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 10:05:31,910][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 10:05:32,413][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 10:05:32,914][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 10:05:33,422][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 10:05:33,926][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 10:05:34,430][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 10:05:34,930][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 10:05:35,433][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 10:05:35,936][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 10:05:36,440][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 10:05:36,948][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 10:05:37,451][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 10:05:37,953][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 10:05:38,456][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 10:05:38,974][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 10:05:39,478][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 10:05:39,983][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 10:05:40,485][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 10:05:40,989][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 10:05:41,498][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 10:05:41,998][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 10:05:42,506][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 10:05:43,010][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 10:05:43,512][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 10:05:44,015][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 10:05:44,516][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 10:05:45,018][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 10:05:45,521][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 10:05:46,020][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 10:05:46,523][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 10:05:47,027][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 10:05:47,528][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 10:05:48,031][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 10:05:48,532][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 10:05:49,036][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 10:05:49,538][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 10:05:50,043][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 10:05:50,556][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 10:05:51,061][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 10:05:51,565][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 10:05:52,071][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 10:05:52,572][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 10:05:53,075][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10105 tokens.
[2025-11-13 10:05:54,001][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.13%, Current % of VRAM taken: 60.39%, Block Peak % of device VRAM: 62.23%, ΔTime: 00:00:33
[2025-11-13 10:05:54,781][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:05:54,783][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:05:54,785][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:05:55,938][__main__][INFO] - Iteration 729 took 1m 7s (45.47% Gen, 52.82% Train). Generation: 30s, Training: 35s. Estimated remaining time: 44h 26m 20s. Estimated total time: 56h 22m 34s. Time estimates for 10 more iterations: 11m 16s, 100 more iterations: 1h 52m 45s, 500 more iterations: 9h 23m 45s.
[2025-11-13 10:05:55,940][__main__][INFO] - Starting iteration 729.
[2025-11-13 10:05:56,445][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 10:05:56,445][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:06:23,831][__main__][INFO] - Number of regex retries in iteration 729: 0
[2025-11-13 10:06:23,831][__main__][INFO] - agents played in iteration 729 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 10:06:24,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:24,633][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:24,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:24,680][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.35%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:24,681][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:06:24,682][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:06:25,485][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 10:06:25,939][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 10:06:26,439][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 10:06:26,935][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 10:06:27,438][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 10:06:27,933][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 10:06:28,431][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 10:06:28,926][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 10:06:29,425][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 10:06:29,926][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 10:06:30,419][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 10:06:30,916][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 10:06:31,412][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 10:06:31,907][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 10:06:32,403][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 10:06:32,897][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 10:06:33,391][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 10:06:33,903][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 10:06:34,404][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 10:06:34,900][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 10:06:35,395][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 10:06:35,895][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 10:06:36,401][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 10:06:36,897][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 10:06:37,401][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 10:06:37,914][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 10:06:38,414][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 10:06:38,914][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 10:06:39,411][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 10:06:39,909][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 10:06:40,417][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 10:06:40,915][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 10:06:41,418][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 10:06:41,919][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 10:06:42,420][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 10:06:42,921][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 10:06:43,421][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 10:06:43,920][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 10:06:44,424][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 10:06:44,932][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 10:06:45,432][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 10:06:45,934][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 10:06:46,435][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 10:06:46,942][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 10:06:47,443][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 10:06:47,945][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 10:06:48,445][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 10:06:48,949][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 10:06:49,453][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 10:06:49,960][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 10:06:50,463][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 10:06:50,972][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 10:06:51,477][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 10:06:51,983][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 10:06:52,486][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 10:06:52,987][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 10:06:53,488][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 10:06:53,987][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 10:06:54,488][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 10:06:54,987][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 10:06:55,487][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 10:06:56,009][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 10:06:56,512][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 10:06:57,014][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 10:06:57,518][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10098 tokens.
[2025-11-13 10:06:58,473][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.02%, Current % of VRAM taken: 60.28%, Block Peak % of device VRAM: 62.13%, ΔTime: 00:00:32
[2025-11-13 10:06:59,240][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:06:59,242][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:06:59,244][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:07:00,284][__main__][INFO] - Iteration 730 took 1m 3s (42.90% Gen, 55.47% Train). Generation: 27s, Training: 35s. Estimated remaining time: 41h 14m 39s. Estimated total time: 53h 11m 58s. Time estimates for 10 more iterations: 10m 38s, 100 more iterations: 1h 46m 23s, 500 more iterations: 8h 51m 59s.
[2025-11-13 10:07:00,286][__main__][INFO] - Starting iteration 730.
[2025-11-13 10:07:00,783][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 72 and human policies 1.
[2025-11-13 10:07:00,784][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:07:33,314][mllm.models.large_language_model_local][WARNING] - Response  Proposal: 10 hats, 10 books, 10 balls did not match regex: (?i)Proposal:\s*((?:\s*(?P<num>(10|[0-9]))\s*(?P<item>hats?|books?|balls?)\s*,?)+), retry 1/3
[2025-11-13 10:07:34,202][__main__][INFO] - Number of regex retries in iteration 730: 1
[2025-11-13 10:07:34,203][__main__][INFO] - agents played in iteration 730 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 10:07:35,048][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:35,072][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:35,094][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:35,116][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.29%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:35,116][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:07:35,117][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:07:36,053][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 10:07:36,510][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 10:07:37,009][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 10:07:37,512][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 10:07:38,010][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 10:07:38,506][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 10:07:39,021][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 10:07:39,517][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 10:07:40,016][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 10:07:40,520][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 10:07:41,018][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 10:07:41,525][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 10:07:42,025][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 10:07:42,527][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 10:07:43,024][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 10:07:43,525][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 10:07:44,026][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 10:07:44,527][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 10:07:45,030][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 10:07:45,543][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 10:07:46,047][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 10:07:46,554][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 10:07:47,057][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 10:07:47,559][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 10:07:48,065][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 10:07:48,566][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 10:07:49,069][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 10:07:49,576][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 10:07:50,078][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 10:07:50,603][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 10:07:51,103][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 10:07:51,603][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 10:07:52,108][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 10:07:52,616][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 10:07:53,125][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 10:07:53,630][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 10:07:54,134][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 10:07:54,661][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 10:07:55,169][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 10:07:55,674][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 10:07:56,178][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 10:07:56,679][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 10:07:57,198][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 10:07:57,699][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 10:07:58,205][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 10:07:58,707][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 10:07:59,209][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 10:07:59,722][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 10:08:00,227][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 10:08:00,727][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 10:08:01,235][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 10:08:01,739][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 10:08:02,244][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 10:08:02,747][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 10:08:03,251][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 10:08:03,757][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 10:08:04,259][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 10:08:04,760][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 10:08:05,265][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 10:08:05,770][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 10:08:06,274][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 10:08:06,778][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 10:08:07,277][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 10:08:07,779][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 10:08:08,281][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10134 tokens.
[2025-11-13 10:08:09,240][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.14%, Current % of VRAM taken: 60.40%, Block Peak % of device VRAM: 62.28%, ΔTime: 00:00:33
[2025-11-13 10:08:09,878][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:08:09,881][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:08:09,883][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:08:11,831][__main__][INFO] - Iteration 731 took 1m 11s (47.04% Gen, 50.22% Train). Generation: 33s, Training: 35s. Estimated remaining time: 47h 13m 55s. Estimated total time: 59h 12m 25s. Time estimates for 10 more iterations: 11m 50s, 100 more iterations: 1h 58m 24s, 500 more iterations: 9h 52m 4s.
[2025-11-13 10:08:11,833][__main__][INFO] - Starting iteration 731.
[2025-11-13 10:08:12,315][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 73 and human policies 1.
[2025-11-13 10:08:12,315][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:08:41,690][__main__][INFO] - Number of regex retries in iteration 731: 0
[2025-11-13 10:08:41,690][__main__][INFO] - agents played in iteration 731 are Alice_buffer, Bob, Bob_buffer, Alice
[2025-11-13 10:08:42,639][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:42,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:42,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:42,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 53.37%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:42,707][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:08:42,708][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:08:43,549][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 64
[2025-11-13 10:08:44,003][mllm.training.trainer_common][INFO] - Processing mini-batch 1 of 64
[2025-11-13 10:08:44,509][mllm.training.trainer_common][INFO] - Processing mini-batch 2 of 64
[2025-11-13 10:08:45,005][mllm.training.trainer_common][INFO] - Processing mini-batch 3 of 64
[2025-11-13 10:08:45,500][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 64
[2025-11-13 10:08:46,022][mllm.training.trainer_common][INFO] - Processing mini-batch 5 of 64
[2025-11-13 10:08:46,517][mllm.training.trainer_common][INFO] - Processing mini-batch 6 of 64
[2025-11-13 10:08:47,016][mllm.training.trainer_common][INFO] - Processing mini-batch 7 of 64
[2025-11-13 10:08:47,513][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 64
[2025-11-13 10:08:48,008][mllm.training.trainer_common][INFO] - Processing mini-batch 9 of 64
[2025-11-13 10:08:48,504][mllm.training.trainer_common][INFO] - Processing mini-batch 10 of 64
[2025-11-13 10:08:49,004][mllm.training.trainer_common][INFO] - Processing mini-batch 11 of 64
[2025-11-13 10:08:49,501][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 64
[2025-11-13 10:08:50,021][mllm.training.trainer_common][INFO] - Processing mini-batch 13 of 64
[2025-11-13 10:08:50,517][mllm.training.trainer_common][INFO] - Processing mini-batch 14 of 64
[2025-11-13 10:08:51,013][mllm.training.trainer_common][INFO] - Processing mini-batch 15 of 64
[2025-11-13 10:08:51,515][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 64
[2025-11-13 10:08:52,019][mllm.training.trainer_common][INFO] - Processing mini-batch 17 of 64
[2025-11-13 10:08:52,528][mllm.training.trainer_common][INFO] - Processing mini-batch 18 of 64
[2025-11-13 10:08:53,031][mllm.training.trainer_common][INFO] - Processing mini-batch 19 of 64
[2025-11-13 10:08:53,534][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 64
[2025-11-13 10:08:54,035][mllm.training.trainer_common][INFO] - Processing mini-batch 21 of 64
[2025-11-13 10:08:54,539][mllm.training.trainer_common][INFO] - Processing mini-batch 22 of 64
[2025-11-13 10:08:55,043][mllm.training.trainer_common][INFO] - Processing mini-batch 23 of 64
[2025-11-13 10:08:55,545][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 64
[2025-11-13 10:08:56,045][mllm.training.trainer_common][INFO] - Processing mini-batch 25 of 64
[2025-11-13 10:08:56,550][mllm.training.trainer_common][INFO] - Processing mini-batch 26 of 64
[2025-11-13 10:08:57,052][mllm.training.trainer_common][INFO] - Processing mini-batch 27 of 64
[2025-11-13 10:08:57,554][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 64
[2025-11-13 10:08:58,057][mllm.training.trainer_common][INFO] - Processing mini-batch 29 of 64
[2025-11-13 10:08:58,557][mllm.training.trainer_common][INFO] - Processing mini-batch 30 of 64
[2025-11-13 10:08:59,061][mllm.training.trainer_common][INFO] - Processing mini-batch 31 of 64
[2025-11-13 10:08:59,561][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 64
[2025-11-13 10:09:00,065][mllm.training.trainer_common][INFO] - Processing mini-batch 33 of 64
[2025-11-13 10:09:00,568][mllm.training.trainer_common][INFO] - Processing mini-batch 34 of 64
[2025-11-13 10:09:01,070][mllm.training.trainer_common][INFO] - Processing mini-batch 35 of 64
[2025-11-13 10:09:01,575][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 64
[2025-11-13 10:09:02,076][mllm.training.trainer_common][INFO] - Processing mini-batch 37 of 64
[2025-11-13 10:09:02,579][mllm.training.trainer_common][INFO] - Processing mini-batch 38 of 64
[2025-11-13 10:09:03,082][mllm.training.trainer_common][INFO] - Processing mini-batch 39 of 64
[2025-11-13 10:09:03,585][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 64
[2025-11-13 10:09:04,085][mllm.training.trainer_common][INFO] - Processing mini-batch 41 of 64
[2025-11-13 10:09:04,595][mllm.training.trainer_common][INFO] - Processing mini-batch 42 of 64
[2025-11-13 10:09:05,095][mllm.training.trainer_common][INFO] - Processing mini-batch 43 of 64
[2025-11-13 10:09:05,601][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 64
[2025-11-13 10:09:06,102][mllm.training.trainer_common][INFO] - Processing mini-batch 45 of 64
[2025-11-13 10:09:06,604][mllm.training.trainer_common][INFO] - Processing mini-batch 46 of 64
[2025-11-13 10:09:07,113][mllm.training.trainer_common][INFO] - Processing mini-batch 47 of 64
[2025-11-13 10:09:07,616][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 64
[2025-11-13 10:09:08,122][mllm.training.trainer_common][INFO] - Processing mini-batch 49 of 64
[2025-11-13 10:09:08,625][mllm.training.trainer_common][INFO] - Processing mini-batch 50 of 64
[2025-11-13 10:09:09,126][mllm.training.trainer_common][INFO] - Processing mini-batch 51 of 64
[2025-11-13 10:09:09,635][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 64
[2025-11-13 10:09:10,136][mllm.training.trainer_common][INFO] - Processing mini-batch 53 of 64
[2025-11-13 10:09:10,640][mllm.training.trainer_common][INFO] - Processing mini-batch 54 of 64
[2025-11-13 10:09:11,139][mllm.training.trainer_common][INFO] - Processing mini-batch 55 of 64
[2025-11-13 10:09:11,640][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 64
[2025-11-13 10:09:12,145][mllm.training.trainer_common][INFO] - Processing mini-batch 57 of 64
[2025-11-13 10:09:12,652][mllm.training.trainer_common][INFO] - Processing mini-batch 58 of 64
[2025-11-13 10:09:13,156][mllm.training.trainer_common][INFO] - Processing mini-batch 59 of 64
[2025-11-13 10:09:13,654][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 64
[2025-11-13 10:09:14,152][mllm.training.trainer_common][INFO] - Processing mini-batch 61 of 64
[2025-11-13 10:09:14,651][mllm.training.trainer_common][INFO] - Processing mini-batch 62 of 64
[2025-11-13 10:09:15,149][mllm.training.trainer_common][INFO] - Processing mini-batch 63 of 64
[2025-11-13 10:09:15,645][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 10026 tokens.
[2025-11-13 10:09:16,539][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 14.04%, Current % of VRAM taken: 60.30%, Block Peak % of device VRAM: 62.03%, ΔTime: 00:00:32
[2025-11-13 10:09:17,283][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:09:17,284][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:09:17,286][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/no_press_10_1_ties_ad_align_nocurrtimestep_seed4321/seed_4321/agent_trainer/trainer_annealing_state.pkl